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ABSTRACT 
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differences in secondary and postsecondary measures, standards, 
assessment, and data collection methodologies; (3) significant 
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for special populations; (5) approximately 38 states receive 
performance measure data from local education agencies; (6) 
approximately half of the state agencies are involved in local 
evaluation processes; (7) the next step for most states is to link 
performance measures to program improvement; and (10) nine 
school ”to-work (STW) states plan to use Perkins Act measures to 
evaluate the success and progress of STW. The appendix contains 
summaries of each state’s performance measures and standards system, 
describing some or all of the following: type of system; measures and 
standards for secondary, post secondary , and special populations; the 
evaluation process; incentives, adjustments, and modifications; and 
future actions. (KC) 
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EXECUTIVE SUMMARY 



System Development: 

► states’ emphasis to date has been on the development of measures and standards 

► 32 states require evaluation of all programs and 17 states require annual evaluation of 
only Perkins-funded programs 

► while approximately half of the states report separate secondary and postsecondary 
systems, almost all states reflect vast differences in secondary and postsecondary 
measures, standards, assessment and data collection methodologies 

► significant factors influencing system development are: 

► state leadership philosophy 

► technology and resources (expertise and funding) available to the state 



Special Populations: 

► 29 secondary and 20 postsecondary state systems include separate measures of 

enrollment, completion, and/or placement for special populations. 



System Implementation: 

► approximately 38 states receive performance measure data from local education 
agencies 

► approximately 50% of the state agencies are involved in the local evaluation 
process 

► approximately 25% of the state agencies are conducting data analysis and 
providing local education agencies with results 

► approximately 25% are providing assessment tools, report forms, guidance and 
procedures to local education units 

► “evaluations” thus far are predominately data collection efforts to determine how 
programs “measure up” against the performance indicators 



Program Improvement: 

► the next step for most states is to link performance measures data to the development 
and implementation of strategies and mechanisms for program improvement 

► 12 states link performance measures and standards to program improvement by 
requiring subrecipients to incorporate the measures and standards data into the local 
application for funds. 



Relationship to STW: 

► a survey of nine STW states indicated they use or plan to use the Perkins measures 
to evaluate the success and progress of STW to serve the needs of their customers. 
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INTRODUCTION 



One of the goals of education is to prepare youth and adults for an increasingly competitive 
and technological workplace. The Carl D. Perkins Vocational and Applied Technology 
Education Act of 1990 has taken great strides toward meeting this objective. Section 1 15 of 
the Perkins Act requires states to develop and implement statewide systems of 
. performance standards and core measures to evaluate the extent to which vocational 
programs meet workplace needs and prepare youth and adults in secondary and 
postsecondary programs for successful entry into either the workplace or continued 
education. Within a broad framework, the law allows states great latitude in selecting 
measures and standards and in determining methodologies for assessing programs that 
best meet states’ educational governance and needs. 

The Office of Vocational and Adult Education (OVAE) is monitoring what states have 
developed and how and to what extent states have implemented their accountability 
system. And while the focus of the requirement is on state and local program improvement, 
the federal office is also assessing to what extent it may tap into this performance data to 
inform and direct future vocational policy and planning. 

Methodology 

The information contained in this paper is extracted from the Annual Performance Report 
submitted by each state to OVAE in December 1993 or with the state plan submitted in May 
1994. Reports were reviewed for the following information: 

► the scope of the system (to whom the measures are being applied), 

► how the system is structured, 

► the types of measures and standards, 

»- how special population groups are being assessed, 

► the nature and extent of system implementation, 

► the types of incentives, modifications, and adjustments provided, and 

► states' plans for continued system development. 



Organization 

This paper is divided into two sections. Section I contains a national snapshot of states' 
development and implementation of their performance measurement systems. Using 
simple statistical analysis, a synthesis and summary information is provided on: 

► the scope and structure of the states' systems, 

► the types of measures and standards developed by states, 

► the factors influencing system development and implementation, 

► how states' systems address the evaluation of special populations, 

► the types of incentives, adjustments, and modifications created or allowed by 
states, and 

► the nature of the data collection process. 

Section II contains a one-to-three page descriptive summary of each state’s performance 
measures and standards system. 



O 

ERIC 



6 



NATIONAL SNAPSHOT 



SYSTEM IMPLEMENTATION 



❖ States’ Approach to the Evaluation Process: 

Implementation of the performance measures system has two phases: (1) an annual 
evaluation of students, programs, and/or institutions against thfe state-developed 
measures, and (2) a program improvement planning process. 



The Perkins Act requires that each recipient of funds annually evaluate the effectiveness 
of their programs based upon the measures and standards. As seen in the chart below, 
there are three major approaches used by states 
to fulfill the annual evaluation requirements. They 

are: State Approach to Evaluation 

Number of States Selecting Each Approach 



Approach #1 : States collect data from each 
local education agency on each performance 
measure, analyze the data and provide the 
local agency with a "report card” of how well 
they have achieved (or not achieved) the 
state standards, 

Approach #2: States provide local 
education agencies with assessment 
instruments, data collection forms, a set of 
procedures, and/or guidance on conducting a 
local evaluation, or 

Approach #3: Local education agencies 
develop their own procedures and conduct 
evaluations against the measures and 
standards as they see fit. 



State Directed 




Locally Driven 
23 



N= 48 - all states do not indicate a 
major approach 



^States’ Involvement in the Annual 
Evaluation Process: 

Findings: 

► approximately one-half of the states are involved in the local evaluation process, 

► approximately a quarter of the states conduct state-level data analysis of local 
programs or institutions and report the findings to the local education agency, 



approximately one-fourth of the states are providing local education agencies with 
assessment tools, data, report forms, and/pr procedural guidance to conduct data 
collection and/or the annual evaluation, 

approximately 86% of the states are receiving information or data from local 
education agencies about the local evaluation (i.e., outcome data, local evaluation 
analysis, or local improvement plans), and 

seven states provided insufficient information to determine if annual evaluations 
are planned or being conducted. 



O Annual Evaluation Process Requirements: 

In addition to requiring annual evaluations, the Perkins Act is prescriptive to the extent 
that it directs a minimum content for the annual evaluation. Annual evaluations are to 
include: 

► the participation of representatives of special populations, 

► identification and adoption of strategies to overcome barriers resulting in 
lower rates of access and success by special populations in vocational 
education programs, 

► an evaluation of the progress being made by special populations members 
in vocational education programs, and 

► an evaluation of the progress that programs are making in providing 
students with strong experience in and an understanding of all aspects of 
the industry. 

States’ emphasis thus far has been on how programs “measure up" on the 
performance Indicators (measures) or against the standards. To what extent the 
evaluation process includes the above components has not been reported. 



O-States’ Approach to the Program Improvement Process: 



The most frequent approach taken by states to reinforce and link the performance 
measures and standards system to program improvement has been to incorporate 
it into the local education agency application for funds. Twelve states report that 
they require sub-recipients to report performance system data and evaluation findings in 
the local application. They are: 



ALASKA 

CALIFORNIA 

GEORGIA 

MICHIGAN 



MINNESOTA 
MISSOURI 
NEW HAMPSHIRE 
NORTH CAROLINA 



PENNSYLVANIA 
SOUTH CAROLINA 
WASHINGTON 
WISCONSIN 
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SYSTEM DEVELOPMENT 



<> System Scope: Who Is Being Measured? 

Graph 2.1 depicts which states require annual evaluations (using the state-developed 
performance measures and standards) of all secondary and postsecondary vocational 
programs and which states require evaluations of only Perkins-funded programs. 



GRAPH 2.1 




As can be seen in the graph, thirty-two (32) states report that they require the 
annual evaluation of all secondary and postsecondary vocational programs 

regardless of whether the program receives Perkins funds. In addition, two states 
(OHIO, PENNSYLVANIA) report that while requiring the annual evaluation of all 
secondary programs they require only those postsecondary programs that receive 
Perkins funds to conduct annual evaluations using the state performance measures 
and standards. 



Seventeen states (17) require the annual evaluation of Perkins-funded programs 
However, while CONNECTICUT and KENTUCKY report that they require only the 



evaluation of Perkins-funded programs, they recommend that all programs be 
evaluated. Of the 17 states which require the more limited evaluation, two states 
(MASSACHUSETTS, NEW HAMPSHIRE) report that sub-recipients receiving Perkins 
funds are required to evaluate all of the vocational programs conducted at the institution 
regardless of whether or not the program area receives Perkins funds. 

Table A identifies which states require evaluation of all programs or evaluation of only 
Perkins-funded programs. 



Table A: State-By-State Listing of Program Evaluation Scope 



ALL PROGRAMS: 



ALABAMA 

ARIZONA 

ARKANSAS 

CALIFORNIA 

FLORIDA 

GEORGIA 

HAWAII 

IDAHO 

ILLINOIS 

INDIANA 

IOWA 

KANSAS 



LOUISIANA 
MINNESOTA 
MISSISSIPPI 
MONTANA 
NEBRASKA 
NEW JERSEY 
NEW MEXICO 
NORTH CAROLINA 
NORTH DAKOTA 
‘OHIO 
OKLAHOMA 
OREGON 



‘PENNSYLVANIA 

RHODE ISLAND 

SOUTH CAROLINA 

SOUTH DAKOTA 

TENNESSEE 

TEXAS 

VERMONT 

VIRGINIA 

WASHINGTON 

WEST VIRGINIA 

* all secondary programs 



PERKINS-FUNDED PROGRAMS: 



ALASKA 

COLORADO 

CONNECTICUT 

DELAWARE 

KENTUCKY 

MAINE 



MARYLAND 

MASSACHUSETTS 

MICHIGAN 

MISSOURI 

NEVADA 

NEW HAMPSHIRE 



NEW YORK 
PUERTO RICO 
UTAH 

WISCONSIN 

WYOMING 



❖System Scope: How Are Systems Being Implemented? 

Another dimension of the scope of a state's system of standards and core measures is 
the time frame or schedule by which states implemented the system. Implementation is 
defined as the direction states have given to districts and institutions regarding of whom 
and when they are to conduct an evaluation of their programs using the measures and 
standards. 
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States report a variety of different implementation models: 

► immediate and full implementation of all measures and standards in 1992-93 
and mandated evaluation of all programs beginning that year (MISSISSIPPI), 

► phase-in of various measures and/or standards as the assessment 
instruments and/or data collection mechanisms are developed (DELAWARE, 
MASSACHUSETTS), 

► an initial pilot study of the measures, standards and the data collection process 
in select programs or institutions and districts with full implementation of the 
system only after reviewing the results of the pilot (MICHIGAN), 

► systematic phased-in evaluation of programs each year until all programs 
are evaluating against all measures beginning in the 1995-96 school year 
(INDIANA), 

► evaluation of only those programs and institutions undergoing state-level 
monitoring during that year (CALIFORNIA), and 

► in a handful of known instances, predominantly at the postsecondary level, the 
required annual evaluations of vocational programs using the performance 
standards and measures have not yet been conducted. 

One is unable to discern from a review of the states’ reports whether one methodology 
has had superior results. It is conceivable that those states which conducted a pilot 
study prior to full system implementation or phased-in measures (as assessment 
instruments and data collection mechanisms became available) have had to dedicate 
less time and resources to troubleshooting gliches in system procedures and have had 
fewer system modifications and refinements than other states. 



❖ Factors Influencing the Scope of the System and Its Implementation: 

A number of factors are reported to have influenced states in determining the scope of 
their evaluation. These include: (1) the regulation process, (2) state leadership 
philosophy, (3) data collection capability, and (4) the states’ philosophy towards 
outcome-based accountability . 



(1) Regulation Process: 

Initially, the draft regulations stated that all programs were to be evaluated. The 
final determination, that states were required to evaluate only federally-funded 
programs, was made late in the regulation process. By this time, some states 
had developed an evaluation methodology based on the draft regulations. 



ERjt 



5 



(2) State Leadership Philosophy : 



In those states where the state agency exerts a strong influence over the local 
education agency, states have established requirements to conduct annual 
evaluations of all programs. In states where the local education agency appears 
to have maximum autonomy, it is not likely that the state has mandated annual 
evaluations of all vocational programs. 

(3) Data Collection: 

The type and scope of the evaluation process are determined by the type of data 
collection process which the state selects for its system of performance 
standards and measures. The type of data collection process selected is 
influenced by two factors: (1) what systems are in place, and (2) in what 
direction the state is moving in developing data collection capability. Many of the 
computerized, state-wide data collection systems already established and in 
operation do not identify programs as federally-funded or non-federally funded. 

In order to tap into these data collection mechanisms, some states chose to 
apply the evaluation requirement to all programs. 

(4) State Philosophy Toward Outcome-based Accountability; 

Those states having a strong commitment towards outcome-based evaluation as 
demonstrated by legislative mandate (e.g., OREGON, ILLINOIS) are likely to 
apply the performance measures and standards to all of their vocational 
programs. 



^System or Systems? 

From a federal perspective, the number of performance standards and measures 
systems and the variation found in them is an important consideration in any discussion 
of developing a national set of performance outcome data. 

Approximately half of the states report having a singular system of performance 
measures and standards for secondary and postsecondary vocational programs while 
the other half describe what appear to be separate systems. Graph 2.2 identifies (1) 
those states which developed a single system of measures and standards for 
secondary and postsecondary vocational programs and (2) those which developed 
separate systems of measures and standards to assess secondary and postsecondary 
vocational programs. 
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GRAPH 2.2 




A closer look at the system of those states reporting a singular system reveals, 
however, that there are significant differences between secondary and 
postsecondary program measures, standards, incentives and adjustments, 
measurement instrumentation and methodology, data collection processes, 
and/or implementation of the system. 

DELAWARE, for example, reports having a singular system of performance standards 
and measures. Academic gain in DELAWARE is defined as “competency in writing and 
mathematics”. How DELAWARE measures academic gain is significantly different at 
the secondary and postsecondary level: 

Secondary Program Level: Academic gain is the difference between the 8th and 
10th grade scores of vocational students on a state-administered test. 

Postsecondary Program Level: Academic gain is the percentage of vocational 
students who complete all communication and mathematics courses required for 
their program as compared to the percentage of all students in the institution 
completing the communication and mathematics course requirements for their 
individual programs. 



As another example, SOUTH DAKOTA also reports having established a singular 
system of measures and standards. However, the measure of program enrollment and 
completion (targeting special population students) is assessed very differently at the 
secondary and postsecondary program levels: 

Secondary Program Level: Program enrollment and completion is the 
percentage of special population students enrolled in and completing vocational 
programs. 

Postsecondary Program Level: The program enrollment and completion measure 
is assessed by identifying and listing the type of supplemental instruction and 
supplementary services provided to special population students and the type of 
affirmative outreach and recruitment efforts used by the institution to recruit 
special population students. 



* 0-Factors in Selecting a System Framework: 

An early issue addressed by states in the development process was whether or not the 
state would develop a single system of measures and standards to assess secondary 
and postsecondary level vocational programs or if it would develop separate systems 
to assess secondary and postsecondary programs. Some considerations in this 
decision-making process included: 

► the cost and resources available to the state to develop and operate the system, 

► state assessment and data collection systems in operation, 

► initiatives and mandates already underway in the state, and 

► the structure of the state delivery system. 

Apart from funding considerations, the predominant factor appears to be the 
structure of the state's vocational education delivery system. Separate systems 
of performance standards and measures are found in 17 of the 25 states where 
the administrative and operational authority for delivery of secondary and 
postsecondary vocational education is delegated to separate agencies. 

However, where a state has established congruent goals, objectives, and priorities for 
secondary and postsecondary level programs, where the responsible agencies are 
communicating information and coordinating their strategic planning, and where the 
state legislature supports performance accountability through state mandate, separate 
education authorities are working closely to integrate the accountability system. The 
Oregon Benchmarking System and the Florida Consensus System exemplify this 
approach. 
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SYSTEM COMPONENTS 



Measures : 



A measure may be defined as "a description of an outcome". The Perkins Act 
allows states great latitude in selecting measures and standards. Section 115 of the 
Act requires only that a state's system include a measure of academic gain, a measure 
of occupational gain, and a measure of performance outcome'in either competency 
attainment, workskill attainment, retention and / or completion of secondary school, or 
placement in employment or further education. 

States have developed many more than the minimally required measures. Graph 2.3 
provides a breakout by program level, depicting the wide range in the number of 
measures developed by states. 



GRAPH 2.3 
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States, in an effort to establish measurement indicators appropriate to their programs 
and to their needs, have produced a wide array of measures. These measures can be 
placed into seven broad categories: 

Academic Skill Measures 
Occupational Skill Measures 
Placement Measures 

Retention, Completion, or Graduation Measures 
General Employability Skills 
Enrollment, Equity, and Access Measures 
"Other" Measures 

However, while the measures may be categorized into these seven areas, it is 
important to understand that within any single category, each state may have 
defined the measure differently. The result is an unlimited number of variations 
on a theme within each category. Graph 2.4 identifies how many states have 
developed secondary and postsecondary measures in a category. 

GRAPH 2.4 
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The graph reflects that states are fairly consistent across program levels in their 
propensity to develop types of measures. As seen in the graph, academic skill and 
placement are the most frequently assessed measures at both the secondary and 
postsecondary level. These are followed, respectively, by occupational skill and 
program retention or completion measures. Measures of enrollment, equity, and 
access appear significantly more frequently in secondary program level 
performance measurement systems (29 states) than in postsecondary program 
level systems (20 states). Placement is the only measure included more often in 
postsecondary systems than in secondary performance measurement systems. 

The next section provides an in-depth look at the differences across states in the 
definitions of measures and the obstacles states encountered in creating specific types 
of measures. Following this is a discussion of the three approaches states have taken 
in developing standards and the broad spectrum of strategies states are using to 
address incentives, system modifications and adjustments. 



Academic Skill Measures: 

Academic skill includes two broad skill areas: basic academic skill and advanced 
academic skill. Basic academic skill may include, for example, communication, math, 
science, reading, listening, and/or writing. Advanced academic skill may include, for 
example, critical thinking or reasoning. States report using the following indicators to 
assess academic skill: 

► gain in student(s) basic academic competencies, 

► gain in student(s) advanced academic competencies, 

► student achievement in basic academic competencies, 

► student achievement in advanced academic competencies, 

► program gain: comparing achievement test scores across years (i.e., 1 1th grade 
student scores compared to 1 1th grade student scores the preceding year), 

► enrollment in or completion of designated academic courses, 

► achievement of SCANS competencies, 

► eligibility for a diploma or degree, or 

► a specified grade point average in designated courses. 

major obstacles: As seen above, the academic skill measure is often defined as 
achievement rather than gain. Although the Act requires states to establish measures 
of academic and of occupational competency gain, not academic and occupational 
achievement, approximately 10-20% of states have not complied with this requirement. 
Major reasons cited by states for not establishing academic gain measures include: 

► lack of authority to increase local assessment requirements, 

► the prohibitive cost associated with increasing testing requirements, 

► lack of appropriate test instruments to measure gain, and/or 

► an inability to incorporate academic skill gain measurement and data 
collection into current (rolling) admissions procedures. 







Occupational Skill Measures: 



States have used one or more of the following indicators to measure an occupationally 
specific skill: 

► student scores on occupational competency tests (pre-post or post-only), 

► student demonstration of a program's occupational competencies as rated by the 
instructor, vocational advisory committee, or technical skill committee and 
documented on a competency checklist, 

► specified course(s) or program completion, 

► student mastery of SCANS competencies, 

► placement (job or continued education, advanced education standing), 

► warranty or institutional guarantee rates, 

► designated grades or grade point average, 

► award of degree, diploma or certificate of skill mastery, employability profile, 

► licensure, industry certification, 

► employer satisfaction , and 

► development of program competencies. 



Eleven states have not established measures of occupational skill for secondary level 
programs and 18 states have not established occupational skill measures for 
postsecondary programs. 

The omission of a measure of occupational skill in vocational education programs may, 
at first glance, appear to be a serious problem. However, all but two state 
postsecondary programs (ALASKA, NEW HAMPSHIRE) have established measures of 
either placement or program completion. States may conclude that program 
completion or placement indirectly assesses occupational competency (i.e., via validity 
generalization -- a placement or program completion or graduation may connote 
occupational achievement). 

major obstacles: The two most predominant obstacles reported by states in 
establishing state-wide measures and standards for occupational competency are the 
lack of standardized occupational competency assessment instruments and the 
lack of standardized assessment standards in a wide range of occupational or 
industry skill areas. 

Although a few states are using commercially developed occupational competency 
tests, occupational competency checklists are the most frequently used instrument for 
measuring and reporting students' mastery of occupational skills. These checklists 
range from state-industry mandated occupational competency checklists to locally- 
developed checklists. Student mastery of a skill is, with few exceptions, evaluated and 
reported by the course instructor. 
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Placement Measures: 



Placement is defined as engagement in post-program activities. Placement indicators 
may target program completers, vocational completers seeking employment or further 
education, or all vocational course takers. States have used one or more of the 
following indicators to measure placement: 

► placement in non-related employment, full-time and/or part-time, 

► placement in related employment, full-time and/or part-time, 

► placement in military, government service, 

► placement in continued education/training, related and/or non-related field of 
study, 

► students' anticipated plans based on a pre-graduation questionnaire, 

► students' expressed desire for employment upon graduation which results in 
employment, 

► students' continued employment 18 months or 2 years after graduation, 

► the unemployment rate of program completers, 

► students' employment status within a given time frame (i.e., 1 month employment 
within the last 9 months), and/or 

► placement in any paid or non-paying activity (i.e., homemaker, volunteer) totaling 
full-time activity. 

Of the performance outcome measures, placement is the most frequently 
established measure. States have established a placement measure in 43 secondary 
and 46 postsecondary state performance measures systems. Placement data may 
include related and non-related employment, further education or training, and military 
service. In 33 states, data on employment and continuing education is aggregated to 
calculate one rate for placement. 

In the remainder of states, the most common refinements to the placement measure 
include: 

► the calculation of separate measures and standards for job placement and for 
continuing education (e.g., HAWAII, ILLINOIS, NEW JERSEY, MAINE, 
MINNESOTA, NEW MEXICO, OHIO, RHODE ISLAND, SOUTH CAROLINA), 

► placement rates calculated separately for special populations (e.g., FLORIDA, 
KANSAS, NEBRASKA), and/or 

► the use of a weighted scale for factoring the placement rate based upon either 
the type of student placement or the placement of a special population student. 
Placement in training-related jobs or the placement of special population 
students are weighted greater than a non-training-related placement or 
placement of non-special population students (e.g., ALABAMA). 



major obstacles: The most significant concern regarding placement data is its 
validity. Most placement data are gathered via student survey. Research supports 
that this type of data collection is less accurate than other methodologies. In at least 
two instances (HAWAII, MARYLAND), placement data are based upon student 
intentions for work or continued education as reported by the student prior to 
graduation. 



Retention, Completion, or Graduation Measures: 

States have used one or more of the following indicators to measure retention, 
completion, or graduation of vocational course-takers, "declared" vocational program 
students, and/or program or course graduates. These measures may refer to either the 
institution, program or course (i.e., retention at the institution or in the program or 
course, or completion of a core of vocational courses for job advancement without 
pursuing a degree or certificate). Indicators being used include: 

► award of a diploma, degree, GED, or certificate, 

► development of an institution or program retention plan, 

► tech-prep enrollments, retentions, completions, 

► dropout rates, 

► graduation rates, 

► student continued enrollment for two or more terms, 

► gainful employment upon completion of a technical institute program, 

► satisfactory completion of a specified level of instruction (enrichment, basic skills, 
or occupational programs), 

► program or institution enrollment at the term’s beginning compared to term end, 

► student transfer to further education or employment, 

► completion rates of Perkins-funded programs vs non-Perkins-funded programs, 
and/or 

► student grades or a specified grade point average of designated course(s). 



General Employability Skill Measures: 

General employability skills are defined as those competencies related to job 
procurement and job retention not associated with specific occupational 
competencies. General employability skills may include, for example: job hunting 
skills, interpersonal relationship skills, teamwork skills, integrity and honesty, creativity, 
self-management skills. States have used one or more of the following indicators to 
measure general employability skills: 

implementation of workplace readiness modules in the curriculum, 
student assessment of skills using standardized tests, 
completion of a course in work ethics, and/or 

instructor assessment of student mastery of workplace competencies. 
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Enrollment, Equity, and Access Measures: 

These measures are most often established by states to address the participation of 
special population groups (including gender) in vocational education. States may have 
used one or more of the following measurement indicators: 

► minimum/maximum enrollment levels per program, 

► special population enrollment proportionate to total school enrollment, 

► special population enrollment proportionate to community population, 

► special population enrollment proportionate by program, 

► special population enrollment proportionate to special population enrollment 
and/or completion in non-vocational programs within the institution, 

► thresholds/ceilings for non-traditional enrollments, 

► a designated instructor/student ratio, 

► the dissemination of information to all groups, 

► proportionate completion and placement rates for special population groups, 

► type of supplemental instruction and supplementary services provided to special 
population students and affirmative outreach initiatives used to recruit, and 

► demonstration of compliance to regulatory requirements. 

Twenty-nine states at the secondary level and 20 states at the postsecondary level 
have established one or more separate measures to evaluate the access, enrollment 
and participation of special populations and/or to assess gender equity in vocational 
education programs. In these instances, data are generally aggregated for the total 
special population rather than separated by individual special population group. 

That approximately half the states are using this approach, does not mean to imply, 
however, that other states are not assessing the participation of special population 
groups. Another option which some other states have selected is to assess the 
participation of special population students as a sub-measure within the data collection 
mechanism implemented for each measure. 

MAJOR OBSTACLES: A major factor in states' capability to assess the participation of 
special populations in vocational education programs is the nature of the data collection 
processes which states are using. Current computerized data collection systems 
may not include special population identifiers, thereby prohibiting states from 
disaggregating data for or by special population group for all measures. States 
report cost considerations and the Perkins Act requirement to include "procedures for 
using existing resources and methods developed in other programs... “ as influencing 
data collection methodology decisions. 

A second major obstacle affecting the assessment of special populations in vocational 
education programs is the ramifications on institutions or programs (real or perceived) 
that do not fully meet the standards established by states: high stakes assessment. 
To address this concern states may have weighted the calculations for members of 
special populations, excluded special population students from one or more measures, 
and/or established different standards for members of special populations. 
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Other Measures: 



The "other" category includes a vast array of measures. Included in this category, 
states have established various measures to assess: 

► vocational awareness, 

► career development or career guidance, 

► student satisfaction, 

► employer satisfaction, 

► earnings, 

► incentives, and 

► program process measures. 



These “other” measures can be grouped into three major categories: 



employer/student satisfaction measures: Eleven states measure student or 
employer satisfaction (via surveys) with either the secondary and/or postsecondary 
programs: 



COLORADO 

IOWA 

MARYLAND 

MINNESOTA 



NEBRASKA 
NEW HAMPSHIRE 
NEW JERSEY 
RHODE ISLAND 



TENNESSEE 
WEST VIRGINIA 
WYOMING 



CAREER GUIDANCE and DEVELOPMENT MEASURES: Career Guidance and Development 
Measures: Career guidance and development measures are included in the following 
state systems: 

ALASKA NORTH CAROLINA SOUTH DAKOTA 

CONNECTICUT OHIO UTAH 

KENTUCKY RHODE ISLAND 



PROGRAM INPUT and PROCESS MEASURES: Process measures include, for example, 
programs or curriculums having identified competencies and state approval, business 
and industry site visits by instructors, collaboration, program productivity, and stable 
program enrollment. States including process or input measures are: 

ALASKA MARYLAND 

ARIZONA MISSISSIPPI 

FLORIDA OHIO 

ILLINOIS RHODE ISLAND 

KENTUCKY SOUTH DAKOTA 



VERMONT 

WASHINGTON 

WYOMING 



It appears that states which are incorporating "other” measures are either: 

► building the state’s performance measures and standards system on to 
and expanding current evaluation and assessment processes so as to 
include the Perkins requirements (this appears especially true at the 
postsecondary level with the significant number of states including 
process measures), or 

► are approaching program accountability from a total quality management 
concept and are assessing the educational process, at various points 
along the process continuum (input, process, as well as, outcome). 



O-Standards: 

Whereas a measure is the description of an outcome, a standard is the level or 
degree to which the student or program is expected to accomplish or achieve 
that outcome. As with the development of measures, states have shown much 
creativity in their approach to establishing standards. States' have established the 
following types of standards: 

Absolute Standards 
Value-added Standards 
Broad Guidelines 
No State Standards 



Absolute Standards: 

Absolute standards are defined as fixed levels of achievement. States have selected 
absolute measures most frequently for measures of academic skill, occupational 
competency, and placement. An example of an absolute standard is: 

"The placement standard for program completers is 70%. " (KANSAS-secondary 
programs 



Value-added Standards: 

Value-added standards are defined as incremental increases in standards based 
upon past performance. States have selected value-added standards most frequently 
for measures of program retention and completion and for those measures of 
program enrollment, equity, and access. An example of a value-added standard is: 

"The graduation rate will increase by a factor of 1.05 each year until a 90% rate 
is achieved." (WISCONSIN- secondary programs) 
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Broad Guidelines: 



Broad guidelines standards are defined as those that enable local agencies and 
institutions to set standards appropriate to their needs and program goals. 
Examples of broad guidelines being used as the state standards include: 

"Program completers will demonstrate mastery of competencies necessary for 
successful entry-level employment in their target occupations and continuing 
their education." (COLORADO) ,and 

"While performance goals (standards) are established by the state, these 
standards refer to the desired aggregate performance of all local systems. 
Standards are not imposed unilaterally to all institutions. Instead, each local 
entity sets standards appropriate to its local situation." (ILLINOIS). 

The establishment of broad guidelines as "standards" is not directed toward any 
particular type of measure. Rather, it appears to be the result of a states' leadership 
style and philosophy. This appears true also for states which set no standards. 



Factors Influencing the Type of Standards Selected by States: 

A number of factors is reported to have influenced states in selecting standards. The 
three predominant factors are: (1) the state's leadership style, (2) the state’s 
philosophy regarding the use of accountability systems, and (3) the state's resources. 

state leadership: In those states where the state agency exerts a strong influence 
over the local education agency, states are more likely to have established 
performance standards. In states where the local education agency appears to have 
maximum autonomy, it is less likely that the state has established rigid standards. 

state philosophy ON accountability: States’ plans for the use of the measures and 
standards evaluation data are also reflected in the states’ approach to standards- 
setting. Where performance measures are used in the implementation of state-level 
program monitoring, states are more likely to establish specifically measurable 
standards which guide them in program evaluation. States emphasizing locally- 
directed program improvement report more development of value-added standards and 
broad guidelines to assist local districts in their program planning and improvement 
process. 

state resources: Those states having access to baseline data through centralized 
data collection are more likely to have established more absolute or value-added 
standards. In states such as OKLAHOMA, OHIO, and NORTH CAROLINA, extensive 
resources, time and money, have been dedicated to developing valid assessment 
systems based upon standards. 




18 



^Incentives, Adjustments, and Modifications: 

• 

The Perkins Act requires that states provide for incentives, adjustments, and/or 
modification in the state system of performance measures and standards to encourage 
service to special populations. Approximately 50% of the states identified 
incentives, adjustments, or modifications in their performance measures system. 
States report that adjustments and modifications to the system measures and 
standards result from local request or local initiative. OHIO and MICHIGAN have 
demonstrated special creativity in identifying incentives to encourage programs to serve 
special populations. 

The following listing identifies a broad spectrum of strategies and initiatives being used 
to fulfill this requirement. States reporting the adjustment, modification, or incentive in 
its system are identified in parentheses. 

ADJUSTMENTS OR MODIFICATIONS: 

► modifications to the measures and standards are allowed in alignment with a 
student's individualized educational plan (ARIZONA, NORTH DAKOTA), 

► local agencies are authorized to modify the State measures or standards based 
upon the criteria identified in the Law, for example, local unemployment rates or 
the demographics of the population (COLORADO, IDAHO, MONTANA, 
OREGON, PUERTO RICO), 

► the exclusion of certain groups of students from the evaluation (MISSOURI), or 

► the use of alternate forms of assessment instruments, equipment, curriculum, or 
educational methods (NEVADA, NEW HAMPSHIRE, OKLAHOMA). 

INCENTIVES: 

► targeting funds to programs with high concentrations of special populations 
(DELAWARE), 

► weighting the calculations (i.e., giving double credit for placement in a job related 
to the training program, giving double credit for the graduation of a student 
identified as an at-risk student (KENTUCKY, VERMONT), 

► state recognition of exemplary programs, for example, those serving high rates 
of special population students or those meeting high standards (MICHIGAN, 
MONTANA, NEVADA, NEW MEXICO), 

► assessing the amount or extent of supplementary services provided to special 
population students (GEORGIA), 

► providing supplementary services to special population students (WYOMING), 

► providing increased staff development opportunities (NEVADA), 

► authorizing interim standards to selected programs or institutions (MISSOURI), 

► increased/reduced reporting requirements (OHIO), 

► giving priority for or withholding funds (OHIO), 

► giving priority for or withholding state accreditation (OHIO). 



SUMMARY: 



States report significant progress in the development of performance measures and 
standards. They report a proclivity to develop measures beyond the minimum requirements 
of the Perkins Act and to apply the system to all programs. Approximately half of the states 
have developed separate systems to evaluate secondary and postsecondary programs. 
Factors having a strong influence on system development are (1) a state's leadership 
philosophy and (2) the resources (expertise and funding) and technology available to the 
state agency. 

Measures and standards established by states fall within seven broad categories. However, 
there are significant differences across states and even program levels in the definition of 
measures within each category. For example, academic skill is defined as a score on a 
standardized reading/math test in one state and it is defined as completion of a course or 
award of a high school diploma in another state. 

Standards fall into three major types: absolute, value-added, and general guidelines. 

Within each of these categories there is broad variance. As with the development of 
measures, states have approached the development of standards within the framework of 
the state's leadership philosophy. States having a strong philosophy of local autonomy are 
more likely to have established broad guidelines or to have deferred standards-setting to the 
local educational agency. Those states having a strongly centralized vocational education 
delivery system are more likely to have set more specific standards. 

The participation of special populations in vocational education is addressed by over half of 
the states via separate measures of enrollment, completion, and/or placement for special 
populations. Data collection systems which do not allow for the disaggregation of data by 
special population are a factor precluding some other states from taking this approach. 

Also, some of the modifications and incentives which states have authorized may have a 
significant influence on the effectiveness of the performance system to promote the 
participation of special populations. The higher the stakes placed on assessment results 
by state or federal entities, the greater is the pressure felt by programs to meet the 
standards: perhaps even at the cost of serving all students. 

States' emphasis and resources have been dedicated primarily to system development. 
States are now beginning to focus on system implementation. System implementation 
includes two components (1) program evaluation, and (2) program improvement planning. 

Approximately 50% of the states are involved in the evaluation process, either conducting 
data analysis or assisting local agencies in the conduct of the local evaluation. Eighty-six 
percent of the state agencies report receiving information or data from the local agencies 
pertaining to the local evaluation. The most frequent approach taken by states to link the 
performance system to local planning is via the local application process. Twelve states 
report requiring local recipients to include evaluation data in their local application. 

States have made significant progress in creating outcome-based accountability systems to 
evaluate the effectiveness of vocational education. However, much work remains in order 
to tie these systems into states' systemic reform for workforce development and to establish 
these systems as an integral component of the vocational education delivery process. 
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ALABAMA 



Type of System: 

Alabama has developed separate measures and standards to evaluate all secondary 
and postsecondary vocational education programs. 

Measures and Standards and Special Populations: 

The secondary measures are: 

high school graduation rate, 
high school exit exam scores, 
placement, 

occupational achievement, 
optimum enrollment, 
teacher/coordinator workloads, and 
curriculum/instruction components. 

Graduation rate measures the percentage of vocational program completers receiving a 
diploma. The rate will improve each year until it reaches the graduation rate of all 
students. High school exit exam scores measure students' academic achievement. 

The final percentage of vocational students passing each part of the graduation 
examination will improve until it reaches or exceeds a passing percentage of 95%. 

Placement measures the rate at which vocational program completers are engaged in 
post program education/employment activities The placement percentage of 
vocational completers will improve each year until it reaches or exceeds 75%. 
Percentage is computed using a weighted scale. Each program completer is counted 
as 1.0. Placements are weighted by type. For example,, full-time, in-field employment 
is weighted as 1 .0 while related employment is weighted as .75 while non-related 
employment is weighted at .5. Completers with disabilities participating in full-time, in- 
field or related employment are weighted at 2.0. The sum of the weighted values is 
divided by the total completer count. 

Occupational achievement is measured using occupational competency tests. The 
average program scores of the occupational preparatory vocational completers will 
increase each year until an average of 70% mastery on a comprehensive test is 
reached and maintained for each occupational program. 

Optimum enrollment measures student enrollment by program. Optimum enrollment 
levels seek to maximize the effectiveness of available funds. The rationale is that 
under-enrollment wastes funds while over-enrollment dilutes optimum instruction, also 
wasting funds. Optimum program enrollments range between 15-20 or 15-25 students 
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depending on the occupational program The standard is for all programs to range 
within +/- 20% of the optimum range. 

Teacher/coordinator workloads measure the number of instructional class periods 
assigned to staff to monitor full-time vocational instruction The standard is that 
teachers and coordinators will be fully involved in only vocational duties/instruction. 

The curriculum and instruction measure monitors that each vocational program 
includes the requirements of the Alabama Performance Accreditation System as well as 
meets the mandates of the Perkins Act. The standard is for a minimum 20% increase 
annually in the number of programs meeting all requirements until all programs meet all 
accreditation requirements. 

The postsecondary measures are: 

competency gain in academic skills, 
implementation of workplace readiness modules, 
student retention, and 

implementation of a placement tracking system. 

Competency gain in academic skills is measured using the ACT-ASSET as a pre/post 
assessment to all students in the associate degree, certificate, or diploma programs. 

Implementation of workplace readiness instructional modules monitors the progress of 
the postsecondary system to provide postsecondary vocational program students with 
instruction in problem-solving, teamwork, and self-management skills. 

Student retention measures the state's progress in developing and implementing a 
retention plan in each institution and the progress of each institution to decrease the 
number of undeclared majors at each college by three percent per year. 

Implementation of a placement tracking system will measure the state's progress 
towards developing and operationalizing a system to identify the rate of program 
completers and program leavers and the percentage of postsecondary vocational 
students entering general or related employment, further education, or military service. 

Evaluation Process: 

The state provided local education agencies with necessary forms to compile the data, 
along with relevant information from State Department of Education records. 

Standards were piloted for one year and, using information submitted from local 
administrators, state staff compiled and analyzed data. Local education units in which 
34% of the programs did not meet a standard were identified and required to prepare a 
program improvement plan. 
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ALASKA 



Type of System: 

Alaska has developed separate systems of performance standards and measures to 
evaluate secondary and postsecondary vocational education programs assisted with 
Perkins funds. 

Measures and Standards: 

The secondary measures are: 

competency-based programs, 
occupational competency, and 
career information. 

Competency-based programs are defined as programs which are outcome-based and 
include job specific competency lists. By the end of 1 994, all courses submitted to the 
State for approval were to be competency-based. 

Occupational competency is measured through student demonstration of competency 
in job specific competencies. All students enrolled in vocational education will 
demonstrate satisfactory progress in achieving mastery of the basic and advanced 
skills specific to each program. 

Career information is a measure of the dissemination of information to students 
regarding opportunities available in vocational education, eligibility requirements, 
specific courses available, potential employment opportunities, and placement services. 
This measure applies to all individuals with special emphasis placed on information 
provided to special population students and their parents or guardians 

The postsecondary measures are: 

competency-based programs, 
student competency gain, and 
access to career planning and placement se; vices. 



Competency-based programs measure the number of courses having outcome-based 
curricula and include instruction which enhances mastery of basic and advanced skills 
specific to each program. By the end of 1996, all curricula will include competencies 
and instruction that reinforce mastery of basic and advanced skills. 
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Student competency gain measures students' mastery of basic and advanced skills 
This is demonstrated through course completion or the results of an assessment 
appropriate for this purpose 

Access to career planning and placement services is measured by an institution's 
development of recruitment information which indicates that placement services are 
available to all program completers. 

Special Populations: 

At the secondary level, guidance is provided to local education agencies in the 
standards and measures system to reinforce and emphasize that special population 
students are to be afforded an opportunity to access vocational programs equal to that 
afforded the general population and outlining that special population students are to 
make satisfactory progress in achievement of basic and advanced skills similar to the 
proportion of general population students making satisfactory progress. 

At the postsecondary level, it is outlined in each standard that special population 
students will receive assistance commensurate with their needs to assure access and 
appropriate progress in their vocational program as well as appropriate career planning 
and placement services. 

Evaluation Process: 

Evaluations are a local responsibility. The local education agency must, however, 
describe in their local application their use of performance measures and standards to 
evaluate progress. Annual performance reports submitted by sub-recipients are to 
include descriptions of competency-based standards and measures used to determine 
if students have met outcomes. They are also to describe program improvement plans 
to ensure that students meet standards and measures. Local program evaluations are 
analyzed by appropriate state staff. The state staff reviews performance standards and 
measures through the monitoring process. 



ARIZONA 



Type of System: 

Arizona’s system of performance standards and measures is the basis of local program 
evaluation for all secondary and postsecondary programs. 

Measures and Standards and Special Populations: 

The system includes five measures. 

basic/advanced academic skills gain, 
occupational competency attainment, 
retention/completion in high school (secondary only), 
placement, and 

incentives (special populations). 

In secondary programs, the academic skills gain measure is targeted toward those 
students who do not possess the identified skills at the beginning of their 
course/program. In this target group, all students are to demonstrate gam in skills 
associated with their program or course. At the postsecondary level, academic skills 
achievement and occupational competency attainment are measured by a program 
completer's attainment of academic proficiency and occupational competencies 
required for his or her program. While at the secondary level, separate standards for 
course completers and for program completers are established to measure 
occupational competency attainment. Eighty percent of course completers will attain 
80% of the occupational competencies associated with the course while 100% of 
program completers must attain 80% of the competencies associated with a program. 
Arizona Curriculum materials that are provided to local education agencies contain all 
the competencies linked to this measure. 

Retention and completion is measured only at the secondary level. It is measured by 
identifying how many students who enroll in a course associated with an approved 
vocational program, continue to attend or complete high school. The standard is 90%. 

Placement, at the secondary level, measuies the number of program completers 
placed into employment, enrolled in training or education at an accredited institution, or 
in military service. At the postsecondary level, placement is measured by the 
percentage of cohort postsecondary and adult students placed in employment, further 
education or training, or the military service. An increase is to be attained each year 
until standards to be established by the state are achieved. 



Implemented in 1994, the measure "incentives to encourage service to special 
populations" establishes that each program, secondary and postsecondary, targeted as 
a recipient of Perkins funds, must provide incentives or adjustments and demonstrate 
that they are in place. 

Evaluation Process: 

Each local district or postseconday institution designates a vocational technical 
education program evaluation coordinator to establish and lead a program evaluation . 
team that includes representatives of special populations, educators, and business and 
industry and to oversee the conduct of the annual evaluation. The results of each 
evaluation are reported to the state. The state developed two guides and conducts an 
annual workshop to assist locals in the understanding and implementation of the 
evaluation process. 

Programs have up to five years to meet fully the performance standards. However, 
after the first year, programs not meeting the performance standards and measures are 
required to develop and implement a local improvement plan. While implementing a 
program improvement plan, programs are designated "provisionally approved". 
Programs not achieving the performance standards for five successive years are 
ineligible to receive state or federal funding. 

Incentives, Adjustments, and Modifications: 

In addition to the measures directed toward targeted groups, the system allows for 
modifications in the achievement of secondary student standards through the Individual 
Vocational Education Plan (IVEP). Also, the retention measurement may be adjusted 
for those students identified as at-risk, using the criteria identified in the Arizona At-Risk 
Project. To measure incentives to encourage service to targeted groups, 
postsecondary programs are to document, annually, their progress in achieving equity 
in the access, retention, and completion of special population students, relative to the 
general population. 
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ARKANSAS 



Type of System: 

Arkansas has established separate systems of performance standards and measures 
to evaluate secondary and postsecondary vocational programs 

Measures and Standards and Special Populations: 

The secondary measures are: 

academic achievement, 
educational attainment, 
occupational competence, and 
access and equity. 

Academic achievement is measured through the administration of pre- and post-tests 
for all students enrolled in Level 1 and Level 2 applied academics courses. The 
Stanford Achievement Test is used to show student gains. 

Educational attainment is a measure of the rate of enrollment in vocational education, 
rate of completion of tech-prep programs, and students' post-high school activities. 

The standard for vocational enrollment is that at least 60% of all students are enrolled 
in at least one vocational course and that there is an increase in the number of tech- 
prep completions. Students' post-high school activities are tracked using a student 
follow-up form mailed some months after graduation. This information is then reported 
as part of the State data system. The standard for placement in further education, 
employment, or military service is to be not less than 80%. 

Occupational competence is measured on all occupational areas for which a 
competency test exists. The standard is an increase in the number of students 
achieving mastery of technical specialty as evidenced by scores attained on the specific 
competency test. In addition, the number of students taking the Tech-Prep Common 
Core courses is tracked. The standard is an increase in the number of students 
completing the complete core of tech-prep foundation courses. 

Access and equity measure the rate of enrollment in vocational education by special 
population group and by race and gender to ensure that enrollment is fair and 
representative of the total population. Special populations enrolling in vocational 
education are to be proportionately representative of the special population of the 
school. Enrollments in vocational courses are to reflect racial and gender equity as 
compared with the total school population. 
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Evaluation Process: 



State Board staff receive annual evaluation reports and determine the degree to which 
local education agencies are making substantial progress in meeting the core 
standards and measures. In addition to evaluating local reports, on-site monitoring of 
local programs is conducted. Finally, each local education agency annually submits a 
program accountability report. 
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CALIFORNIA 



Type of System: 

The system originally adopted by California in 1992 was determined to be 
unnecessarily complex and burdensome. The current system was approved in January 
1994. California has developed separate definitions for performance standards and 
measures at the secondary and postsecondary levels to evaluate all secondary and 
postsecondary vocational programs. Each measure has both an absolute and a value* 
added standard. 

Measures and Standards: 

The system includes the following measures: 

basic and advanced academic skills, 

retention, and 

placement. 

At the secondary level, basic and advanced academic skills are measured by student 
performance on the state-wide California High School Performance Assessment in the 
tenth grade. The absolute standard is derived from obtaining a score of 75 on a 100 
point composite index comprised of rated factors which include student mobility, level of 
parent education, and the percent of students receiving Aid to Families with Dependent 
Children (AFDC). At the postsecondary level, the standard and measurement of 
academic skills is the state-wide average of students successfully completing a 
vocational education course that integrates academics into the curriculum and leads to 
a certificate, degree, or diploma. 

To measure retention at the secondary level, the dropout complement rate (percentage 
of students who did not drop out in grades 10-12) is calculated. The standard is 
achieved by scoring 75 on a composite index or improving the dropout complement 
rate by 5 percentage points over the base year. Retention at the postsecondary level is 
measured by calculating the percentage of course completers against the state-wide 
average. 

Both the secondary and postsecondary levels measure placement by the percentage of 
students employed, enrolled in higher education or additional training, or entered in the 
military. The standard is the state-wide average. 

Special Populations and Evaluation Process: 

The system includes information on each special population group for all measures. In 
addition, at the college level, data on course and program completion rates, including 
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student grades, placement and transfer rates is monitored for each special population 
group, as compared with the total vocational education population in general. 

At the secondary level, annual program evaluations conducted by districts also include 
monitoring of access, success, and progress of students who are members of special 
populations. The results of the annual evaluation, including rates of access, success, 
and progress, are described in the local agency's application for funding. 
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COLORADO 



Type of System: 

Colorado has developed a singular system of performance standards and measures to 
evaluate all approved, Perkins-funded, secondary and postsecondary vocational 
education programs, not solely those funded under Title II, Part C. 

Measures and Standards: 

The nine measures are: 

achievement of basic/advanced academic skills, 
attainment of general occupational skills, 
attainment of specific occupational competencies, 
successful program persistence and completion, 
high school graduation or its equivalent (secondary only), 
placement, 

client satisfaction (postsecondary only), 

sex/ethnicity equity, and 

equal access for special populations. 

Achievement of basic and advanced academic skills measures students' mastery of the 
foundation skills outlined in the Secretary's Commission on Achieving Necessary Skills 
(SCANS) Report. These include the basic skills, thinking skills, and personal qualities. 
Vocational completers will demonstrate, using locally selected assessments, mastery at 
level sufficient to attain successful entry level employment and continuing education. 

Attainment of basic occupational skills measures students' mastery of the workplace 
competencies used by effective workers. These are the competencies identified in the 
SCANS Report and include allocation of resources, interpersonal skills, acquisition and 
evaluation of information, understanding of systems, and technology. Program 
completers will demonstrate, on locally selected assessments, mastery of these 
competencies at levels necessary for successful entry level employment in their target 
occupations and continuing their education. 

Attainment of specific occupational competencies is measured using locally selected 
assessment in the specific occupational cluster related to the training program. 

Program completers will demonstrate mastery of at least minimal occupational skills 
and other prerequisites necessary for entry level employment in their target occupation 
or industry. 

Program persistence and completion measures, by institution, the rate at which 
students successfully complete a defined vocational program. Recipients are to specify 
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a self-defined target of program persistence and completion of at least 80% or five 
percent above the 1991-92 baseline completion rate, whichever is less 

High school graduation measures successful completion of secondary school or its 
equivalent. Each eligible recipient will specify a self-defined target for high school 
completion of at least 90% or five percent above its 1991-92 baseline rate, whichever is 
less. 

Placement measures the rate at which program completers enter additional education . 
or training, military service, or employment. The local recipient identifies a self-defined 
target for placement of at least 95% or at least five percent above its 1991-92 baseline 
placement rate. 

Client satisfaction is a postsecondary measure. The measure calculates the 
percentage of clients (current students, program completers, and employers) satisfied 
with the vocational education program. Local recipients determine their target client 
satisfaction rate which is to be at least 90% or five percent above the 1992-93 baseline 
rate of client satisfaction. 

Sex/ethnicity equity measures the rates at which measures 1-7 are achieved by various 
sex and ethnic groups. Any observed unfavorable gender or ethnic differences in 
attainment of the standards or in program enrollment will steadily diminish. Whenever 
a group is under-represented, the local recipient is to identify a target participation rate 
which is at least five percent above the 1991-92 baseline rate or a rate proportional to 
the group's representation in the school's population, whichever is less. 

Equal access for special populations measures the rates at which measures 1-8 are 
achieved by various special population categories. The proportion of special 
populations learners enrolled in vocational education programs will steadily increase. 
Target participation rates will be at least five percent above the 1991-92 baseline rate 
or in proportion to their representation in the school population, whichever is less. 

Special Populations: 

Measure #9, equal access for special populations, is established to address the 
expanded participation of special populations in vocational education programs. In 
addition, for students for whom attainment of the standards applied to measures 1-3 is 
an unrealistic expectation within the timeframe of their program participation, 
individualized goals (standards), representing significant increase from their baseline 
assessment, may be developed. 
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Evaluation Process: 



Each recipient is required to submit an annual self-evaluation to the state. All local self- 
evaluations are reviewed by the Board to determine that substantial progress is being 
made in meeting the standards and measures. In addition to the self-evaluation of 
program outcomes as measured by the performance standards and measures, 
Colorado continues its Quality Vocat i o n a l Pr o g ram Rev i ew and Improvements Proce ss 
(Q.ProcessV Together these evaluations (input/process and impact/outcome) provide 
information as to the cause of certain outcomes and guidance for program 
improvement. 

Incentives, Adjustments, and Modifications: 

Modifications to the nine measures and standards may be proposed based on local 
economic, geographic, and demographic factors. They may include adjustments 
consistent with the unique or specific goals of programs to meet the needs of target 
groups or special populations. Individualized standards may be developed for specific 
preparation programs for handicapped students who anticipate entry into a supported 
or sheltered work environment, consistent with the goals of their individualized 
education plan. 
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CONNECTICUT 



Type of System: 

Connecticut has developed a singular system of performance standards and measures 
requiring the self-evaluation of all secondary and postsecondary programs receiving 
Perkins funds. Additionally, the state recommends that an eligible recipient assess all 
their vocational programs whether or not these programs or students receive benefits 
under the Perkins Act. 

Measures and Standards and Special Populations: 

The system contains both generic performance measures and standards and program- 
specific measures. Generic performance measures assess the performance of school- 
wide vocational programs while program-specific measures are considered to be the 
core competencies for each program area. The measures are. 

vocational program awareness, 

nontraditional enrollment, 

vocational program access (special populations), 

placement, 

relevancy of program to labor market need, 
basic academic skill, 
advanced academic skill, and 
employability skills. 

The standard for vocational program information awareness is that a process be in 
place which informs all students of the full range of vocational programs available in the 
comprehensive high school, regional vocational-technical school, and the regional 
agricultural center. The target population for this measure is the 8th grade student. 

Non-traditional enrollment is measured by the number of students enrolled in courses 
considered non-traditional for their gender. The standard is an annual increase in the 
proportion of nontraditional students. 

Placement is measured by the number of available program completers either 
employed, in further education or training, or in military service. The standard is an 
annual improvement in the numbers. Within five years all vocational programs are to 
be assessed vis-a-vis their relevancy to labor market and societal needs as well as 
programmatic needs (i.e., professional in-service and equipment needs). This action is 
the standard for determining the relevancy of the program to labor market need. 

Basic academic skill is measured by the percentage of 10th grade students in 
vocational education who pass the Connecticut Academic Performance Test. This rate 
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is to equal the percentage of total students in the school passing this 10th grade test 
Advanced academic skills are measured using the Connecticut Common Core of 
Learning. These competency lists and guides identify the vocational, mathematics, 
science, and communication competencies by program area and course as well as the 
standards to be achieved through Grade 12. The standard for employability skills is 
that all vocational students acquire employability skills taught by the school through a 
choice of curricula (i.e., World of Work or Workplace Readiness). 

The measure of vocational program access assesses the enrollment rates of special 
needs students in vocational education programs. The percentage of special needs • 
students enrolled in each vocational education program will approximate the 
percentage of special needs students in the total population of the school. 

Evaluation Process: 

The state provides local recipients with evaluation forms, procedures and technical 
assistance to conduct the self-evaluation. Self-evaluation reports are then submitted to 
the state. Local recipients determine whether or not substantial progress towards 
achievement of the standard is being accomplished. 

Incentives, Adjustments, and Modifications: 

In addition to, and as part of the self-evaluation process, local recipients are to identify 
adjustments and incentives designed to encourage the successful participation of 
targeted populations in applied (vocational) programs. 
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DELAWARE 



Type of System: 

Delaware has developed a singular system of performance measures and standards 
for evaluating secondary and postsecondary vocational education programs receiving 

Perkins funding. 

Measures and Standards: 

The system includes both measures of learning and competency gain in academic skills 
and measures of performance attainment. They are: 

competency in writing, 
competency in mathematics, 
competency in science (proposed), 
school completion, and 

competency attainment in vocational programs (proposed) 

Competency in writing and mathematics is used as the measure of gam in basic and 
advanced academic skills. At the secondary level, this is assessed using the Delaware 
Transitional Assessment administered statewide to grades 3,5,8, and 1 0. Data on 
students in approved vocational programs will be disaggregated from total data. The 
standard is the profile gain of vocational programs between grades eight and ten. This 
profile will match or exceed the profile for all students. At the postsecondary level, this 
is assessed by calculating the percentage of vocational program students completing 
all written communication and mathematics requirements as compared with the 
percentage of students completing these requirements in all institutional programs. 

The percentage of students in vocational-technical programs completing all 
requirements will exceed the institutional percentage. 

The competency in science measure will be adopted when the state-appointed Science 
Commission completes its work on math and science curriculum frameworks. 

School completion measures the rate at which vocational program students graduate 
from school as compared with the graduation rate for all students. The graduation rate 
for vocational program students will match or exceed the graduation rate for all 
students. 

Competency attainment addresses the recommendations contained in the Secretary's 
Commission on Achieving Necessary Skills (SCANS) Report. This measure will be 
phased in as state-mandated frameworks for integrating academic and vocational 
education are developed. 
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Evaluation Process: 



Each local recipient is to conduct an annual evaluation which encompasses either the 
entire local agency population or a representative sample of participants. Data on 8th 
grade achievement test scores was collected in 1993. The 10th grade scores, to be 
used as a basis for the state-established baseline, will not be collected until Spring 
1995. Graduation data collected in 1993 was unable to be disaggregated by vocational 
program students and is not useable. Therefore, state data collection has yielded no 
information, to date, to identify areas for local improvement. 

Incentives, Adjustments, and Modifications and Special Populations: 

Local recipients are authorized to adopt programmatic adjustments which allow 
students with special needs the opportunity to meet the standards set for all vocational 
education students. In addition, the restriction of the use of funds areas with the 
highest concentrations of special populations is perceived as the system incentive. 
Funds are targeted only toward those programs in the upper half of all district 
vocational programs as ranked by special population. 
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FLORIDA 



Type of System: 

Florida's Consensus System of Vocational Program Performance Measures and 
Improvement Targets includes measures and targets (standards) for evaluating all 
secondary postsecondary, and adult vocational education programs. The Consensus 
system includes both "accountable" measures and informational measures to serve as 
a complete set of program improvement tools. 

Measures and Standards and Special Populations. 

The accountable measures include: 

productivity, 
completer placement, 
program earnings, 
institutional earnings, 
basic academic skills, 
advanced academic skills, and 
enrollment (special populations). 

Productivity is applied only to postsecondary and adult vocational programs. 

Productivity is defined as the rate at which all "accountably enrolled students are being 
productively served by their program. This is measured by determining the percentage 
of program completers and "placed" program leavers against the total program 
enrollment. Program productivity rates are calculated for both all students and targeted 

population students. 

Completer placement rates are determined for all program levels, (secondary, 
postsecondary, and adult) by calculating the rate at which all vocational program 
completers are placed in related employment, further training, or military service. Rates 
are determined for all students and for special populations. 

Program earnings and institutional earnings are the average quarterly earnings levels 
of former program participants as reported to the Department of Labor and 
Employment Security. Institutional earnings averages are calculated for all students 
and again for special population groups. These measures are applied only to 
postsecondary and adult vocational education programs and institutions. 

At the secondary level, basic skills are measured by the percentage of vocational 
program completers who are eligible for or receive a high school diploma. For adult 
vocational programs, it is the percentage of students who receive vocational certificates 
for completion of a 450 + hour program and who have passed a basic skills test in 
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mathematics and language At the postsecondary level, the measurement of basic 
skills acquisition is the number of students receiving an associate in science degree. 

The advanced academic skills measure is applied only at the secondary level. The 
measure of advanced academic skills is determined by calculating the percentage of 
vocational program completers who pass two or more high-level math, science, or 
communications courses. The state has rated all courses in a three tier framework. 
Exceptional students and those in dropout prevention programs are excluded from the 
calculation. 

Enrollment, the rate at which targeted populations students enroll in vocational job 
preparatory programs compared to their enrollment in all vocational programs in the 
school, is measured only for adult and postsecondary programs. 

The targets (standards) are variable indices in that they are based on previous years 
attainments. The targets for the productivity, placement, and earnings of special 
populations are the same as those for all students. 

Evaluation Process: 

The consensus system was designed to satisfy the requirements of all state and federal 
mandates regarding performance measures for vocational education programs in an 
integrated, singular system. The system was constructed around existing databases 
rather than increasing the data collection burden. The state collects and analyzes the 
data, then electronically transmits the results (by measure) back to the local agency. 
This reduces the burden on the local district or institution and insures consistency in 
computation methodology. To implement the Consensus system, the state drafted a 
comprehensive implementation manual and conducted a series of workshops. The 
results of the Consensus System measures, along with other indicators, are published 
in the Florida School Report each year. 

Future Activities: 

Future activities include the writing of a series of technical assistance papers to further 
support the use of the system at the local level. In addition, the state will implement a 
new state-level program review system that enhances the relationship between output 
and outcome measures with their standards and the educational process that affects 
them. This effort will form the basis for program improvement planning. 
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GEORGIA 



Type of System: 

Georgia has developed separate systems of performance measures and standards to 
evaluate all secondary and postsecondary vocational education programs. 

Measures and Standards: 



At the secondary level, the measures are: 



academic gain, and 
performance. 

The state has established three options from which sub-recipients may select to 
measure academic gain. Sub-recipients may measure academic gain by either 
calculating the score gain of vocational students in areas of verbal and quantitative 
competencies on the Iowa Test of Basic Skills (ITBS) and Test of Achievement and 
Proficiency (TAP) or equivalent instrument between 8th grade and 12th grade, or by 
identifying the percentage of vocational students who progress to the next grade level 
at the end of the school year. For all options, a sliding scale of gain is applied as the 
standard. 

The state has established four indicators from which sub-recipients may select to 
measure performance: school retention/completion, competency attainment, life and 
employment skill attainment, and projected placement. School retention/completion 
measures the percentage of vocational students completing each grade. The standard 
is a sliding scale increase until a 91+% is maintained. Competency attainment 
measures the percent of occupationally specific competencies attained by 100% of the 
students based upon teacher assessment. The average percent of competencies 
achieved is to be 85% or more in each course. Life and employability skill attainment 
also measures the percentage of competencies attained. The standard has not been 
established. Projected placement is a measure of students anticipated post-graduation 
plans based upon the results of a pre-graduation questionnaire. The standard is a 
sliding scale increase until the percentage of students projecting placement is equal to 
or greater than 70%. 

The postsecondary measures are: 



academic gains, 
occupational skills attainment, 
job or work skill attainment, 
retention, and 
placement. 





21 



0 

ERIC 



47 



At the postsecondary level, academic gain is determined by calculating the score gain 
of diploma/degree-seeking students in areas of verbal and quantitative competencies 
on the TTAP (ASSET will be used beginning in 1994) or equivalent instrument in pre- 
post-test administrations. 

Occupational skills attainment is defined as an institution's rate of warranty claims. A 
positive outcome is a total institutional rate less than 2%. Job or work skill attainment 
is measured by the rate at which diploma/degree graduates are provided with 
instruction in work ethics and receive one or more quarterly grades for such 
performance on their transcripts. The standard is for 1 00% of all students to receive 
such instruction and performance evaluation. 

Retention rate is defined as the rate at which program exiters graduate and the rate at 
which institutional exiters graduate. The standard is that 35% of all program exiters will 
graduate and 35% of all institution exiters will graduate. 

Placement is calculated for each program in an institution. The standard is that 85% of 
all program graduates are employed in the field for which they trained or a related field, 
are continuing their education, or are in military service. 

Evaluation Process: 

Using evaluation guides and procedures established by the state, local recipients 
conduct local evaluations and forward the results of the evaluation to the State 
Department of Education. State staff review and determine the major dimensions of 
impact assessment and trends. 

Incentives, Adjustments, and Modifications and Special Populations: 

At the postsecondary level, measures and standards are established to evaluate the 
extent and type of special support services offered and provided to members of special 
populations. The outcomes are measured for members of special populations. 
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HAWAII 



Type of System: 

Hawaii has developed separate measures and standards for evaluating all secondary 
and postsecondary vocational education programs. 

Measures and Standards: 

The five measures developed for the secondary level are: 

academic achievement, 

course competency attainment, 

placement - employment, 

high school completion, and 

placement - continued education/military service. 

To meet the standard for academic achievement, 75% of vocational program students 
will pass the State-mandated HSTEC by the completion of Grade 12. Course 
competency is measured by calculating the percentage of students completing course 
requirements as determined by instructor assessment. The standard established is 

80%. 

The standard for high school completion is for 75% of vocational education students to 
receive a high school diploma or its equivalent prior to their 18th birthday. The 
standard is calculated by the number of vocational education students graduating in 
proportion to the number of vocational education students in their senior year. 

Placement into employment is measured by calculating the percentage of students 
informing their counselors of a desire to obtain employment upon graduation actually 
obtaining relevant and meaningful employment. The standard is 50% employment 
upon graduation. Placement into continued education or military service is determined 
by the percentage of students who respond to a follow-up survey who are in the military 
or some form of postsecondary education. The standard is 45% of those responding to 
a follow-up survey. 

The measures established for postsecondary programs include. 

course completion — remedial courses, 
course completion - general education courses, 
course completion — vocational courses, 
retention, and 
placement. 
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The measures of course completion target different levels of academic skill acquisition 
or vocational competency. Attainment of the standard is determined by the rate of 
students who complete courses with a passing grade as measured by the credits 
earned ratio. The standard for remedial/developmental courses is 60%, for general 
education courses it is 65%, and for vocational education courses it is 70 /o. 



Retention is measured by the percentage of students retained in their program of study 
for two or more terms. The standard is 50%. 

Placement includes attainment of employment in a job related to the training program, 
enrollment in additional education or training, and entrance into military service. At the 
postsecondary level, the standard is 70%. 



Evaluation Process: 

Data is collected from the school by the Department of Occupational Education, 
Occupational Development Section through the school's "Report of Progress To Date" 
submitted on or before November 15th of each year. First year evaluations were 
conducted during 1 993. 

Incentives, Adjustments, and Modifications: 

After review of the first year evaluation results, the state has determined to retain the 
same measures and standards for another academic year. Any need for modification 
or revision to the measures or standards will be revisited after the second year 
evaluation. 
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IDAHO 



Type of System: 

Idaho has developed separate systems of program performance measures and 
standards to evaluate all secondary and postsecondary vocational education programs 

in the State. 

Measures and Standards and Special Populations. 

The ten secondary program measures fall within three major areas: 

occupational skill attainment, 
academic skill attainment, and 
equity and access. 

Occupational skill attainment is measured using two indicators: (1) students mastery of 
occupational competencies listed in the Idaho curriculum (or other comparable list) for 
the occupational area and (2) placement rates from the student's initial program. The 
standard for mastery of occupational competencies is that a minimum of 90% of the 
students meet 80% of the competencies. Placement standards are a 90% positive 
program termination rate and a 75% employment or continuing education within one 
year of graduation. 

Five measures address the attainment of academic skills. The first measure compares 
the percentage of vocational completers receiving a "C" average in 14 core academic 
courses with the percentage of all students in the school receiving a C average. The 
second measure assesses students' writing skills using the Direct Writing Assessment. 
Using a scoring rubric from 1-5, the established standard is three. The third measure 
of academic skill attainment compares vocational students' scores on the Test of 
Achievement and Proficiency (TAP) with the average scores of all students. The 
standard is that the average score of all vocational students is equal to or greater than 
the average scores of all students. The last two measures assess academic gain by 
comparing the difference in student scores on the Direct Writing Assessment given in 
the 8th and 1 1th grades and by comparing the difference in student scores on the Iowa 
Test of Basic Skills (ITBS) given in 8th grade with students' scores on the TAP given in 
1 1th grade. The standard is that the TAP scores will average 21 points greater than 
the average ITBS scores. 

Three measures are used to assess equity and access for all students: the calculation 
of the instructor/student ratio; the dissemination of vocational education information; 

and the development of student career/educational plans. The standard for 

instructor/student ratio is 1-to-300. The standards for dissemination of information and 
the development of student career plans by the end of ninth grade are 100%. 
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Twelve postsecondary measures fall within four major areas. 

basic/advanced academic skills, 
occupational skills attainment, 
labor market indicators, and 
accessibility and access (special populations). 

Three measures address academic performance: the number of programs in which 
75% of the students had a 2.0 GPA in communication and math courses; the number of 
programs in which 75% of the students had completed the required academic courses; 
and the percentage of institutions where students scored at least 1/4 of a standard 
deviation higher on a pre- post-test. 

Occupational skill attainment is measured through completion rates and student grade 
point averages. The standards are a 75% program completion rate across all programs 
and a 70% GPA rate of 2.5 or more across all programs. Program completion is the 
measure used to assess short-term training programs. The standard is an 85% student 
completion rate. 

The labor market indicators used to evaluate programs are placement rates. The 
standards are a 90% positive termination rate (placement in any employment, 
education, or military service), and a 75% placement rate in training-related 
employment or continued education. 

Three measures are designed specifically to address the progress of special 
populations: the percentage of special population students achieving a 2.0 GPA as 
compared with all students; the academic course completion rate; and the program 
completion rate for special populations. The standards are for the percentages for 
special populations to be equal to that of the general student population. 



Evaluation Process: 

Local education agencies are required to conduct annual evaluations. The State 
Division of Vocational Education collects data on all vocational-technical programs as to 
the school's performance on the measures and standards. 

Incentives, Adjustments, and Modifications: 

Local recipients may modify the measures and standards, based upon the factors and 
conditions outlined in the Act, through the local two-year plan. 
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ILLINOIS 



Type of System: 

Illinois has established a system for evaluating all secondary and postsecondary 
vocational education programs that is aligned with the six state goals for vocational 

education. 

Measures and Standards and Special Populations: 

The six measures are: 

academic achievement, 
occupational competence, 
employment, 
educational attainment, 
economic competitiveness, and 
access and equity. 

Academic achievement is measured at the secondary level by assessing the 
percentages of students attaining a level 2 score on the IGAP reading and math 
assessment. At the postsecondary level, academic achievement is measured by using 
program completion rates. 

At both the secondary and postsecondary levels, occupational competence will be 
based upon assessment of workplace skills, applied academics and technical skills 
assessment. Assessment instruments are still under development. In addition, 
postsecondary institutions calculate percentages of students activating the institutional 
guarantee and are returned for remediation to denote occupational competence. 

Educational attainment is measured at both the secondary and postsecondary level by 
determining the percentage of program completers and/or the percentage of program 
completers continuing into further education or training. 

At both the secondary and postsecondary level, two measures are used to determine 
employment outcome: employment and employment retention. Employment is 
calculated for the second quarter after high school graduation or, at the postsecondary 
level, for the third quarter following the end of the academic year. Employment 
retention is indicated by continued employment into the sixth quarter following high 
school graduation or, at the postsecondary level, continuous employment for four 
quarters after program completion. 

Economic competitiveness is a secondary level measure of a program's productivity, 
quality of service, and training costs. Information is derived from state studies. 
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Access and equity are measures calculating the enrollment and participation of special 
populations in vocational education courses at both the secondary and postsecondary 
level. The enrollment standard is special population enrollment in vocational education 
courses in comparison to its representation in the total school population. 

While standards (performance goals) are established by the state, these standards 
refer to the desired aggregate performance of all local systems. State goals are not 
automatically imposed unilaterally on all institutions. Instead, each local entity sets 
goals appropriate to its local situation. 

Evaluation Process: 

The State Board oversees the evaluation of all programs. Local recipients annually 
evaluate programs using the state's macro-data-synthesis system. Performance 
information is provided to each local recipient by the state on all measures for all 
vocational-technical education students in the cohort, special populations in the cohort, 
all students in targeted programs and special populations in targeted programs. A 
detailed special population report is also provided which reflects performance 
information for each special population category. 

Future Activities: 

Future activities include the phase-in of the measures of occupational competence, 
postsecondary education continuation, and employment retention. Future activities 
also include the identification of skill standards for selected occupations, development 
of applied academics and work skills assessments and state-wide testing, incorporation 
of existing certification/credentialing results in the system, the design of a study of 
economic competitiveness measures, and the coordination of the performance 
measurement system with other workforce preparation programs. 




28 



INDIANA 



Type of System: 

Indiana has developed separate systems of performance standards and measures to 
evaluate all secondary and postsecondary vocational education programs at institutions 
receiving Perkins funds. Beginning in 1995-96, the system of performance standards 
and measures will be applied to all vocational and technical education programs 
regardless of funding source. 

Measures and Standards and Special Populations: 

The secondary system is composed of four measures. They are: 

academic competency gain, 
occupational competency attainment, 
retention, and 
placement. 

Academic competency is measured by the percentage of program completers who 
increase their scores or achieve a grade level equivalency score in English, language 
arts and mathematics on a pre-post test. The standard is 80% for all students. 
Accommodations (i.e., time, readers) are identified to ensure that all special 
populations students are included in the assessment process. 

The standard for occupational competency is for at least 85% of program completers to 
demonstrate mastery in the occupational skills necessary to obtain employment. When 
the attainment of standardized measures (i.e., SOCAT) or competency-based 
instruction is not appropriate for a special population student, attainment of the goals 
established in the student’s individualized education plan (IEP) is used. 

Retention is measured by the number of vocational education students completing all 
requirements for a diploma. For special education students, completion of the 
certificate of completion described in the IEP is counted toward the 85% standard. 

Placement includes employment, continued education, or military service within one 
year of the projected graduation date. Special education students who are receiving 
transition services, and who are unemployed, are not counted in the first year cohort 
group. However, placement of special population students in sheltered workshops or in 
supported employment is counted in the placement rate. 
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The postsecondary system also contains four measures They are: 

access and equity, 

academic achievement, 

occupational competency attainment, and 

placement. 

Each measure addresses adjustments for special population students where deemed 
appropriate. Students identified by a secondary IEP, or determined to be unable to 
meet the basic competencies for vocational education programs are exempt from 
measurement. 

Access and equity entails enrollment, completion, and placement rates for targeted 
populations The standard is a gain standard requiring a five percent rate increase in 
each area until the rate is equal to that of the overall or non-targeted population. 

Academic achievement is measured by the number of students entering the institution 
who require remedial preparation. The standard is for an annual five percent increase 
in the percentage of students successfully completing these requirements in three 
semesters or less (until a rate of 90% is achieved). The measurement established to 
determine occupational competency attainment is a five percent annual increase (until 
100% rate is achieved) in the percentage of program completers proficient in the skills 
for the occupation they are prepared to enter. 

The placement measure applied to postsecondary programs differs from placement in 
the secondary system in three areas: (1) followup is based upon the status of program 
completers within six months of program completion, (2) employment must be in an 
occupation related to the training program, and (3) the established standard is a five 
percent annual increase until a 70% rate is achieved. 

Evaluation Process: 

Indiana state law requires the Indiana Commission on Vocational and Technical 
Education to conduct an evaluation at least annually that assesses all technical 
education programs offered at the secondary and postsec jndary levels. All data 
collection methods are in place to gather all information. To facilitate local 
implementation, the state has developed packets for local institutions that include 
information on the system, an implementation schedule, and state agency contact 
persons. 
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IOWA 



Type of System: 

Iowa has developed a single system of performance standards and measures to 
evaluate all secondary and postsecondary vocational education programs. 

Measures and Standards and Special Populations: 

The eight measures are: 

CORE MEASURES (required): 

gain in basic/academic skills, 

gain in specific occupational competencies, 

occupational competency attainment, and 

access. 

OPTIONAL MEASURES (select 1 of 3): 

student satisfaction before graduation, 
student satisfaction after graduation, or 
retention. 

FUTURE (select 1 after implementation of initial measures): 

placement, or 
employer satisfaction. 

The methodology for measuring gain in academic skill and in occupational competency 
is left to the local district. However, the state-established standard is that 80% of 
program completers achieve the standard. 

Occupational competency attainment is measured by the number of program 
completers who attain 100% of the program competencies determined critical for 
employment. The local district is charged to identify the critical competencies, including 
workplace competencies, and the level of competency necessary for employment. 

Student satisfaction prior to graduation is measured by students’ perceptions of their 
preparation. Factors to be rated include: job skill development or improvement, goal 
achievement, positive work attitude development, occupational and academic 
competency development, and educational foundation for further learning. Local 
institutions are to develop methodology and instruments appropriate to their needs. 

Student satisfaction after graduation with their occupational and academic skill 
preparation measures the perceptions of vocational program completers who are 
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employed in a related field. The standard is that 90% of the respondents will rate their 
preparation satisfactory. 

Retention is measured at the secondary level by the percentage of students who enroll 
in the second unit of a vocational program and, subsequently, complete the program. 
The standard at the postsecondary level is that 80% of the students who enroll in a 
vocational program with the intent of completing it, do complete it 

The measurement of placement includes employment (related and unrelated to 
training), further training, and military service of program completers within one year of 
graduation. The standard is 90% and only those program completers not seeking 
employment, training, or military enlistment are excluded from the calculation. 

Employer satisfaction is measured by identifying employers perceptions of vocational 
program completers in their employ in the following areas, producing quality work, 
observing job-related rules, working effectively with others, demonstrating skills 
necessary to do the job, demonstrating knowledge of the industry, demonstrating 
potential for advancement and/or increased responsibility, and displaying initiative. The 
standard is that 90% of the employers will rate completers as satisfactory or better. 

The access measure is defined as the enrollment levels of all students, including those 
who are members of special population groups and members from diverse racial/ethnic 
groups in each vocational education program. The student enrollment by targeted 
group within each program is to reflect, within a 10% variance, the enrollment of the 
population in the district or institution or progress towards that standard In addition, 
data for the core measures and the optional measures will be reported by gender, racial 
and ethnic group, and category of special populations 

Two processes are associated with each measure; a definitive improvement process 
which sets absolute standards and a continuous improvement process which enables a 
district or institution to demonstrate progress towards attainment of the absolute 
standard. Locals may decide which process to implement. 

Evaluation Process: 

Evaluation is conducted locally. Both the data results and a narrative description of the 
methodology and instrumentation used in the evaluation process are forwarded to the 
state in an annual report. 
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KANSAS 



Type of System: 

Kansas has developed separate systems of performance standards and measures to 
evaluate secondary and postsecondary vocational programs. While all measures were 
implemented during the 1992-93 school year after a pilot of selected schools was 
conducted, the scope of the measures and/or standards will be expanded dunng a three 
year phase-in period. This phase-in helps to ensure that no undue burden is placed on the 
schools during the initial years and allows time for the competency profiles to be developed- 
by the Kansas Competency-Based Curriculum Center at Washburn University, Topeka. 

Measures and Standards: 

The secondary measures are: 

learning gains in basic and advanced academic skills, 
competency attainment in workplace skills, 
specific occupational competency attainment, 
retention or completion of secondary school, and 
placement. 

During the first year of implementation (1992-93) learning gains in math were measured 
using the Kansas State Assessment Instrument or another norm-referenced test of the 
school’s choice. In year two, the measure was expanded to assess gains in language arts 
(reading, comprehension, and writing). This was expanded further with the addition of 
science in year three. Pre-post testing is conducted at least annually and the pre-test is 
administered no later than the 1 1th grade. 

Competency attainment in workplace skills currently entails measuring: mastery of 
managing resources, team participation skills, using computers, and decision-making. is 
will be expanded to include interpreting and communicating information, developing of 
learning strategies, exhibiting integrity and honesty, creative thinking, and using self- 
management skills. Individual schools are to develop procedures to determine competency 
in these areas. The standard established by the state is that 80% of all vocational program 
completers will master a minimum of 80% of these competencies. 

Specific occupational competency is measured by the program instructor against 
competency profiles and competency checklists. These data are reported annually to the 
state. The standard is the same as that for workplace skills. Beginning in the second year 
of system implementation, local advisory committees will validate program competencies 
(100%) and update checklists every three years. Annual employer satisfaction surveys will 
be added in year three to assess the specific occupational skills and competencies of 
program completers employed in areas related to their training. The standard is for 80% of 
all program completers employed in areas of their training to receive satisfactory ratings. 
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The measurement for retention or completion of secondary school is the high school 
diploma. The graduation rate will equal or exceed that of the general population from that 

school. 

Placement is measured by calculating the ratio of program completers holding jobs, 
pursuing further education or military services as identified through an annual follow-up 
survey. The standard is a 70% placement rate. A placement rate for special population 
students in the secondary program is also calculated. If placement rates for these groups 
falls more than 10% below that of non-special population students, the institution is to 
develop a plan for improvement. 

The postsecondary system measures include: 

learning gains in basic and advanced academic skills, 
competency attainment in workplace skills, 
specific occupational competency attainment, and 
placement. 

In the postsecondary system, learning gains in academic skills based upon a pre-post test 
are measured for only those students enrolled in remedial courses. Ninety-five percent of 
those students completing a remedial course will show an increase in score. A form has 
yet to be developed to aggregate this data to the state. 

The measures of competency attainment in workplace skills, specific occupational 
competency attainment, and placement are the same as those at the secondary level. 

Special Populations: 

Both the secondary and postsecondary systems include a measure and absolute standard 
for ensuring that the needs of special population students are identified and that they are 
provided opportunity to receive necessary supplementary services. Criteria, data collection 
forms and reporting systems are being developed to document attainment of this measure. 
Data on special population participation by program area are currently available. 

Evaluation Process: 

The Kansas State Board of Education has developed software for recording and reporting 
data which will permit electronic reporting of data to the state agency and computer 
aggregation and manipulation of the data. The software is designed to enable institutions 
to evaluate the measures of competency attainment in workplace skills, retention or 
completion of high school, and the needs of special population students at the institutional 
. level. Measures of academic skills and specific occupational skill competency can be 
evaluated by institution or program. The first data collection for the Kansas statewide 
system of measures and standards of performance began in October 1993. 



KENTUCKY 



Type of System: 

Kentucky has developed a system of performance standards and measures by which 
recipients are to evaluate their secondary and postsecondary vocational education 
programs receiving Perkins funds. The State Board recommends that recipients use 
the system to evaluate all secondary and postsecondary programs regardless of 
whether or not the program receives Perkins funds. 

Measures and Standards and Special Populations: 

The system includes nine measures, most of which contain multiple indicators. 



basic/advanced academic skill, 
guidance & counseling, 
occupational competency, 
completion, 
placement, 

collaboration between agencies, 
enrollment (special populations), 
gender mix, and 
retention. 



At the secondary level, basic/advanced academic skill is measured by the performance 
levels (novice apprentice, proficient, and distinguished) attained by 12th grade 
vocational students as reported on the Kentucky Instructional Results Information 
System in writing, reading, math, science, and social studies. At the postsecondary 
level, the percentage of vocational students maintaining satisfactory grades in remedial 
or developmental courses is used to measure basic academic skills while the 
percentage of vocational students maintaining satisfactory grades in other academic 
courses is used to measure advanced academic skills. 



That guidance services are provided to all vocational students is demonstrated at the 
secondary level by an assessment of student's learning style, use of interest surveys, 
dissemination of career awareness information, completion of career exploration 
activities, and the development of career plans based upon assessment and career 
goals At the postsecondary level, the indicators for assessing the guidance and 
counseling measure are that an articulation process is in place with secondary career 
guidance programs, that career counseling services are available at the postsecondary 
level, that counselors participate in professional development activities, and that a 
system of career information is maintained. 
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At both the secondary and postsecondary level, occupational competency is measured 
using student grade point averages. The percentage of vocational education program 
students with less than a 2.0 grade point average is compared to the average grade 
point average of all students in vocational courses. Also counted at the secondary 
level, is whether or not student competency records are completed for each student in 
an occupationally specific program. 

The standard for program completion is a variable rate based upon all students who 
exit programs except for transfers. The standard is to increase each year until it 
reaches or exceeds a rate to be determined by the state. 

Placement at the secondary and postsecondary level is measured by calculating the 
rate of program completers who make a successful transition to additional training or 
education, military service, or work. The standard is a value-added standard of a 5% 
increase each year until 90% is attained. 

Collaboration is measured at both levels. The standard is that documentation of 
collaborative activities between vocational education and other agencies must be 
maintained. At a minimum, agency programs with which collaboration is expected are: 
Job Training Partnership Act programs, Vocational Rehabilitation programs, Adult Basic 
Education and General Education Diploma programs, and Community Based 
Organizations. 

In both secondary and postsecondary institutions, a 1 % - 4% increase over a four-year 
period in non-traditional enrollment by program in a minimum of 75% of the vocational 
programs is the measurement to determine gender mix. A gender neutral program is 
one in which the ratio is a minimum of 75/25 percent. 

Retention is measured by the rate of graduation from high school of vocational 
students. The graduation rate will equal or exceed that of the general school 
population. No measure of retention is established at the postsecondary level. 

Access to vocational education by students who are members of special populations 
(disabled, disadvantaged, limited English proficient) is determined through the 
enrollment measure. The standard established for both the secondary and the 
postsecondary level is that the percentage of special population students enrolled in 
vocational programs be proportionate to the percentage of special population students 
in the general student population. This information is aggregated to the institution level. 

Incentives, Adjustments, and Modifications: 

In the system, adjustments are provided for by weighting the calculations for special 
population students in the completion and placement measures. 
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LOUISIANA 



Type of System: 

Louisiana has established a singular system of performance measures and standards to 
evaluate all secondary and postsecondary vocational education programs. 



Measures and Standards: 

basic and advanced academic skill gain, 
competency and job/work skill attainment, 
program completion, and 
placement. 

Academic skill gain measures the increase in academic skill attainment by measuring 
students' successful progression through a sequence of academic core courses or by 
measuring the increase in the number of students successfully completing proficiency 
examinations in basic and advanced course skills. At the secondary level, assessment of 
student gain in academic skill is determined using the math, English and language arts, 
composition, science and social studies components of the Louisiana Education 
Assessment Program (LEAP). At the postsecondary level, technical institutes are to use 
the Test of Adult Basic Education (TABE) while community college and university associate 
degree programs will assess academic gain using the assessment instrument selected by 
the Institute of Higher Education. The standard for all programs is that they reflect an 
annual increase in these numbers. 

Competency and job/work skill attainment measures the increase in occupational skills by 
measuring students' successful progression through a sequence of academic and 
vocational course skills. Competency is determined by demonstrating an understanding of 
all aspects of an industry and occupation specific skills in at least one occupation through 
all written and performance assessments prescribed by the sequence of courses. 

Program completion measures the number of students completing a sequence of academic 
and vocational courses resulting in the award of a diploma, certificate of achievement, 
associate degree, or (in technical institutes only) gainful employment. The standard is an 
increase in the percentage of students achieving these goals. 

Placement measures the number of program completers who enter additional education 
and/or training, employment, or military service. The standard is for an annual increase. 

Evaluation Process: 

Local assessment and report of the gains as they relate to the standards and measures are 
conducted annually. Recipients of federal funds determining that they are not achieving 
substantial progress will develop a local improvement plan. 
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MAINE 



Type of System: 

Maine has established a singular system of performance measures and standards to 
evaluate secondary and postsecondary vocational education programs receiving 
Perkins funds. 

Measures and Standards: 

The measures are: 

high school graduation, completion, or recovery, 
linked-postsecondary program enrollment, 
successful school-to-work program completion, 

mastery of basic/advanced academic, employment, industrial, and occupational 
skills, 

successful entry into appropriate employment, 
related occupational/industrial placement, 
entry wage differential/advanced placement status, and 
three-year positive change of employment status. 

High school graduation, completion, or recovery measures the percentage of students 
receiving high school diplomas, certificates of completion or GED's. The standard is 
that at least 90% of vocational students should graduate with their class. The 
remaining 10% should, within a three-year window, return to school and graduate, 
receive a GED, or complete an adult education program. 

Linked-postsecondary program enrollment measures the number of secondary 
vocational program and school-to-work system students enrolling in articulated 
postsecondary courses of study. The standard is that 80% of the school-to-work 
system students will matriculate to postsecondary programs within six to nine months of 
high school graduation. 

Successful program completion applies to school-to-work system students. 

Discounting migrants, the student attrition rate in applied technology and school-to- 
work opportunity programs at both the secondary and postsecondary levels should not 

exceed 1 5% per year. 

Mastery of academic and occupational skills assesses the attainment of skills by 
program completers. All successful completers will master the minimum competencies 
and other prerequisites necessary for entry-level employment in their chosen 
occupations, industries, or career areas. This will be determined by the number of 
individualized itemized, verified Certificates of Skill Mastery awarded. 
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Placement assesses the percentage of vocational program completers who transition 
into appropriate employment, enroll in further education, or enter military service. The 
standard is for 90% to be placed within six to nine months of program completion. 
Appropriate employment is either competitive, supported or sheltered employment in a 
career area offering reasonable wages and fringe benefits, safe and satisfactory 
working conditions, and realistic prospects for personal growth and career 
advancement. The standard for related occupational/industrial placement is 60% 
placement of program completers in a career directly related to their field of study. At 
least one half of the remaining completers should enter appropriate employment in an 
unrelated area. 

Entry wage differential will compare the wage and advanced placement of vocational 
program completers to those entry-level workers without comparable education, 
training, and preparation. The standard, a three-year positive change of employment 
status, will be calculated by aggregating wage and employment data for three years in 
order to compare the hours of work, increased wages and earnings, increased 
responsibility within a single job, and job promotion or other career advancement of 
vocational program completers with those not receiving equal career preparation. 
These measures will be implemented with implementation of the Automated 
Occupational and Labor Market Information System (AOUS). 

Evaluation Process: 

Each local recipient conducts a self-evaluation using the standards and measures. 



Incentives, Adjustments, and Modifications and Special Populations: 

Local modifications are authorized based upon the factors and conditions outlined in 
the Perkins Law. They may include adjustments consistent with the unique or specific 
goals of programs designed to meet the needs of target groups or special populations, 
including adults requiring short-term training or retraining. Individualized standards 
may be developed for special preparation programs for students with disabilities who 
anticipate employment in supported work or sheltered work environments, consistent 
with their individualized educational plan. Local modifications may be developed using 
the "regression model" method developed by the Department of Labor which generates 
unique standards for JTPA service areas. 
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MARYLAND 



Type of System: 

Maryland has developed separate systems of performance standards and measures to 
evaluate all secondary and postsecondary vocational education programs conducted by 
recipients of Perkins funds. 

Measures and Standards: 

The seven measures comprising the secondary system are: 

skill competency attainment, 
postsecondary status, 
postsecondary performance, 
assessed student knowledge, 
student attainment, 
postsecondary decisions, and 
student participation. 

The measure of skill competency attainment may be demonstrated through written 
examination or the awarding of a competency certificate or employability profile. The state 
has established no standard as yet. 

Postsecondary status is a measure of the rate of employment and continuing education of 
high school graduates. Within one year of graduation, the percentage of high school 
vocational graduates employed or continuing their education is to equal (satisfactory) or 
exceed by +1.0% (excellent) the employment rate for the local educational agency service 
area. 

Postsecondary performance is student success as determined by employer satisfaction or 
postsecondary educational standing. To be satisfactory based upon employer satisfaction, 
reporting employers must rate a minimum of 92% of employed completers employed by 
them as meeting or exceeding minimum job requirements. A standard for "good standing" 
is on hold awaiting the collection of baseline data. 

The assessment of student knowle Jge is determined from the percentage of students 
passing the Maryland Functional Tests (reading, mathematics, writing, citizenship) or the 
percentage of students achieving satisfactory performance levels on each of the criterion- 
referenced tests in the Maryland School Performance Assessment Program. 

The measure of student attainment refers to high school completion. Indicators for this 
measure include the percentage of students who complete course requirements that qualify 
them for admission to the University of Maryland and the percentage of high school 
graduates completing an approved Career and Technology Education Program. 
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The measure, postsecondary decisions, identifies the relationship between students high 
school program and expressed plans on graduation. Information is collected annually by 
the state using the High School Graduate Follow-up Questionnaire completed by 
graduating seniors 30 days prior to graduation. The state encourages local agencies to 
conduct follow-up studies to verify this information. 

Multiple variables are tracked to measure student participation, including yearly dropout 
rates and average daily attendance (under 5 days and over 20 days). 

The five postsecondary measures are. 

program status, 
skill attainment, 
completer status, 
completer satisfaction, and 
employer satisfaction. 

Program status is determined by program enrollments over a two year period. The 
standard is for enrollment by program to not decrease by more than 10% and by more than 
10 students since the previous year. 

Skill attainment is measured by the number and percentage of program completers 
receiving degrees, licensure, or certification. The established standard is that the 
percentages do not decrease by more than 10%. 

Completer status is a measure of the rate of employment and the continuing education of 
degree program or certificate graduates. Within one year of graduation, the percentage of 
graduates employed or transferred to a four year institution is to equal the employment rate 
for the local area and that 85% of the employed program completers are to be employed in 
jobs related to their educational program. This measure is assessed every other year. 

The measurements of program completer satisfaction and employer satisfaction include an 
assessment of both the academic and the occupational skill preparation received in the 
educational program. The standard for these measures is that 90% of surveyed students 
and employers each rate the preparation as satisfactory. 

Evaluation Process: 

In the secondary vocational education delivery system, data for the performances 
measures are collected by the 24 local school systems and forwarded to the state level via 
the reporting procedures of the Maryland School Performance Program. Data are analyzed 
and results are reported back to the local education agency in report card form. The 
postsecondary measures are incorporated into the Program Data Monitoring System 
(PDMS), established 14 years ago. Data are collected at the State level, analyzed, and the 
results provided to the 18 community colleges for program improvement. 
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MASSACHUSETTS 



Type of System: 

Massachusetts has developed a single system of performance standards and 
measures to evaluate secondary and postsecondary programs receiving Perkins funds. 
While the system utilizes the same measures for both secondary and postsecondary 
programs, it includes two sets of standards - one set for secondary programs and one 
set for postsecondary programs. Currently, the performance standards and measures . 
are applied to only those programs receiving Perkins funds. Over the next three years, 
however, the state plans to phase in the requirement that school districts and colleges 
receiving Perkins funds must evaluate all Perkins-eligible programs whether or not they 
are supported with Perkins funds. 

Measures and Standards: 

The system currently includes the following measures: 

completion, 
placement, and 
follow-on placement. 

Completion may be measured at the secondary level by either (1 ) identifying the 
percentage of individuals enrolled in a program who have successfully completed their 
annual program of studies and are ready to advance to the next level of coursework or 
the workplace, or (2) identifying the percentage of senior year students who will 
graduate with both a high school diploma and a certificate of mastery in an 
occupational area. Completion is measured at the postsecondary level by identifying 
the percentage of students who have completed a one-year certificate program within 
two years or who earn a two-year certificate or degree within four years. The standard 
has been established at 90% for secondary programs and 50% for postsecondary 
programs. 

Placement is measured by identifying the percentage of program graduates placed in 
the workforce, further education or the military within four months of graduation. Two 
sets of standards are employed at the secondary level - one for placement in training- 
related employment (70%) and one encompassing both training-related and unrelated 
employment (90%). At the postsecondary level, however, only training-related 
employment is recognized and the placement standard is 90%. The validity of this 
response-driven placement report is reinforced by basing placement rates on program 
graduates only and by requiring programs to ensure a 65+% response rate. 

A follow-on survey of program graduates is conducted approximately one year after 
graduation. At the secondary level, rates are adjusted to 60% and 90% respectively. 
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At the postsecondary level, programs must show an 85% placement rate to meet the 
standard. Again, validity is reinforced by requiring a minimum 65% response rate. 

Evaluation Process: 

Beginning in December 1993, schools began reporting program completion rates, 
placement rates, and follow-on placement rates to the state on the annual Standards 
and Measures Report utilizing data reported each October on the Occupat i ona l 
Fdnratinn Report In the initial report, locals also compared their rates against state 
standards and set goals for each program for each year 1995-1998 (by 1998 all 
programs are to reach state-established standards). In December 1994, schools 
submitted their second annual Report. This report compares rates against State 
standards as well as their locally-established interim goal for 1995. Any program not 
reaching its yearly goals is to submit a local improvement plan with this annual Report. 

The exception to the reporting process described above is completion data for public 
two-year colleges. The Department of Education will use data collected for the Right- 
To-Know Law to measure completion rates and data will be examined for the first time 
in 1994. 

System refinement activities being conducted by the state include a collaborative effort 
between Massachusetts Job Council, University of Massachusetts-Amherst, and the 
State Department of Education to develop and implement the MISER system to track 
placement and one-year follow-on of program completers through the wage-reporting 
system presently maintained by the Massachusetts Department of Revenue. 



Future Activities: 

Future activities to be conducted by the State include: (1 ) the redesign of data 
collection instruments so that data can be extracted for various special population 
groups and tech-prep program participants, and (2) development and implementation 
of competency gain and competency attainment measures. Toward this end, the State 
Department of Education and Northeastern University have identified core 
competencies for the 30 occupational areas having the majority of student enrollment. 
As the next step, they plan to develop a process to measure competency gain and 
attainment. Unresolved to date however, is the development and implementation of a 
measure of academic competency gain and academic competency attainment. 
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MICHIGAN 



Type of System: 

Michigan has developed separate systems of performance standards and measures to 
evaluate secondary and postsecondary vocational programs receiving Perkins funds. 



Measures and Standards: 

The secondary system has three measures: 

academic achievement or gain, 

placement in jobs or continuing education, and 

content competency {in Life Management Education (Consumer & Homemaking 
Education) only}. 

Academic achievement is measured by determining the percentage of career and 
technical education students meeting the requirements for a high school diploma. 
Placement rates are determined by the Office of Career and Technical Education using 
a program completer follow-up survey. 

The postsecondary system includes measures in three areas. 

basic academic skills attainment, 
advanced academic skills attainment, and 
attainment of occupational work skills. 

Basic academic skills attainment is measured by identifying the percentage of 
academically under-prepared students successfully completing a developmental 
education course. Advanced academic skills attainment is measured by identifying the 
percentage of students successfully completing general education courses. 

Attainment of occupational work skills may be measured by either determining the 
number of students successfully completing an occupational specialty course, 
occupational program, or by determining the percentage of program completers who 
are employed or who are enrolled in additional education, training or enlisted 'n military 

service. 

When the percentages are calculated for each of the postsecondary measures, a 
sliding scale of continuous improvement is then applied to determine whether the 
program has met the standard. 
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For example: 



If the percentage of students successfully completing a developmental course is 
between 75% - 80%, the amount of annual gain should be at least four percent. 
Once 100% is achieved the benchmark must be maintained. 

Special Populations: 

In the postsecondary system, the measures of advanced academic skills attainment 
and occupational work skill attainment include sub-measures that breakout the 
percentage of special population students meeting or exceeding the standard. 

Evaluation Process: 

Through 1 994, performance standards and measures at the secondary level are being 
piloted at only eight secondary school districts or schools as demonstration sites. 

Postsecondary institutions are required to include the 1991-92 and 1992-93 standards 
and measures data and analysis in their local Perkins Plan application submitted in May 
1994 to the state. Local improvement plans, if applicable, are to be implemented 
during 1994-95. 

Incentives, Adjustments, and Modifications: 

At the postsecondary level, recognition will be given for an exemplary concept, 
program, or professional service rendered to members of special populations. The 
"Carl Perkins" Merit Award is an annual method for the State Board of Education and 
the Community College Board to recognize individual schools. Nominees and 
recommendations are submitted to both Boards for review and vote. The award is 
presented at the annual TRENDS Conference. 

Future Activities: 

At the secondary level, future proposed activities include the development of an 
implementation manual and a video tape on implementing performance standards and 
measures, conducting workshops on implementation procedures, pilot-testing 
competency attainment measures for Career and Technical Education wage-earning 
programs, coordinating with the Michigan Education Assessment Program (MEAP) to 
modify the MEAP to use as the academic gain measurement instrument, and state- 
level monitoring of the local evaluation process and implementation of local program 
improvement plans. 
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MINNESOTA 



Type of System: 

Minnesota has developed separate systems of performance standards and measures 
for evaluating all secondary and postsecondary vocational education programs. 

Measures and Standards: 

The secondary system includes five measures: 

participation, 

graduation, 

placement, 

employer satisfaction, and 
student satisfaction. 

Participation is measured by program area and special population status using 
MINCRIS. This instrument can be modified to collect graduation rates as well as other 
data elements (i.e., sex equity parity within program areas). 

The Minnesota Secondary School Follow-up System is used to collect placement, 
employer and student satisfaction information. The attributes measured by employer 
satisfaction include technical knowledge, work attitude, work quality, and the overa 
education of the employee. Data collection for the Class of 1 991 provided first year 
data for the system. 

The postsecondary system has implemented six effectiveness measures. 

enrollment by college (special population), 
enrollment by program (special population), 
graduate satisfaction, 
related employment, 

related employment (special population), and 
retention. 

A state-set standard has been established for only one measure - placement. The 
standard is that a program maintain a 60% training-related placement rate for two 
consecutive years. Inability of a program to meet this standard results in state review 
for program retention, restructure, or closure. For the remaining measures, local 
institutions are to establish "reasonable” standards, using a weighting methodology that 
allows for a comparative, composite score for each program. 
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Evaluation Process: 



Data on each of the measures is collected using existing data systems. For some, 
modifications to the data systems are being phased-in to support the performance 
measurement system. Local applications describe the locally established standards. 
Using the results of the local self-evaluations, annually the state targets two or three 
programs for review by a focus team. 

Future Action: 

Future postsecondary initiatives include implementation of three additional measures of 
occupational competency: industry-based accountability, graduate grades of all 
students, and graduate grades of special population students. 
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MISSISSIPPI 



Type of System: 

Mississippi has developed a singular system of performance measures and standards to 
evaluate all vocational education programs. 

Measures and Standards: 

The eight measures are: 

program process, 

enrollment, 

retention, 

completion, 

graduation, 

placement, 

occupational competency attainment, and 
academic gain. 

Program process measures the extent to which vocational programs accomplish the 
applicable Level I and Level II indicators as listed in the appropriate Mississippi Program 
Review Instrument. Review Instruments are developed for administration, instructional 
programs, vocational guidance, special populations programs, community-based 
organizations, single parent programs, sex equity programs, tech-prep consortia, and 
short term adult vocational programs. The standard is that programs must accomplish 
100% of Level I and 70% of the applicable Level II indicators. 

Enrollment measures the average enrollment per full time equivalent teacher unit 
(student/faculty ratio) of a program. The standards are: enrichment programs -- 1 1 
students/1 FTE; basic skills programs - 85/1; secondary occupational programs -- 30/1; 
postsecondary occupational programs - 1 5/1 ; short-term adult programs -- 1 2/1 . 

Retention measures the percentage of students who have completed and satisfactorily 
passed a level of instruction. The standards are: enrichment programs - 87%; basic 
skills programs - 84%; secondary occupational programs - 84%; postsecondary 
occupational programs — 86% and short-term adult vocational program — NA. 

Completion is a measure of the percentage of students who completed the program. 
Standards of 33%, 29%, and 80% are established for secondary programs, 
postsecondary programs, and adult classes, respectively. Graduation is a measure of the 
percentage of students who received a diploma, certificate, or degree. The standards 
are: secondary programs —20% and postsecondary programs —18%. 
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Placement is a measure of the percentage of students who completed or exited basic 
skills programs, or secondary or postsecondary occupational programs and are 
considered positively placed. The standards are 40%, 38%, and 47% respectively 

Occupational competency attainment measures the percentage of program completers 
demonstrating satisfactory occupational competencies on either a standardized exit exam 
(preferably a certification or licensure test) or on an individual student competency profile. 
Eighty-five percent of program completers will demonstrate occupational competency. 

Academic gain measures the percentage of students who have completed a level in the 
program (pass or fail) and have demonstrated gain in the basic and advanced academic 
skills. Academic gain can be measured by one of three methods: (1) passing academic 
courses at each level of instruction in the program [two courses at each level for 
secondary programs and one course at each level for postsecondary programs], 2) 
completing an Individualized Education Program (IEP), or (3) showing a gain score on a 
norm-referenced, pre/post-test series. The standard is 93% for enrichment, basic skills, 
and secondary occupational programs and 45% for postsecondary programs. No 
standard is established for adult vocational programs. 

Evaluation Process: 

Local districts are to summarize the results of individual program evaluations on forms 
provided by the state. The state has developed a computer software program to assist 
local districts in calculating the data. Summary sheets are forwarded to the state by 1 
October each year. Local improvement plans as required are forwarded to the state with 
the local evaluation summary. Local improvement plans are required for programs not 
attaining minimum effectiveness. For supportive services, programs, and projects this is 
defined as meeting the program review measure. For instructional programs this includes 
the standards related to placement, occupational skills, academic skills, and a majority of 
the other measures and standards. The state will review all evaluation results, conduct 
desk audits and monitoring visits to verify data. 

Incentives, Adjustments, and Modifications and Special Populations. 

Local districts may request exceptions to meeting the standards when extenuating 
circumstances exist which prevent a program from meeting the standard. Examples of 
extenuating circumstances include unusual rates of transfers from a district or the 
opening of new programs. Adjustments to the standards and the establishment of local 
standards are calculated to reflect the enrollment of special populations in programs and 
the unemployment rate for the local district as compared with the state baseline. The 
result may be lower local standards when the percentage of special populations or the 
unemployment rate is higher than the state baseline or higher standards when the rate of 
special populations or unemployment rate is lower than the state baseline. No 
adjustments are authorized for graduation, occupational competency, or academic gain 
measures and standards nor for short-term adult programs. 
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MISSOURI 



Type of system: 

Missouri has established a singular system of performance standards and measures for 
evaluating all secondary occupational training programs and all postsecondary 
preparatory vocational education programs [500+ hour adult programs, associate 
degree and certificate] benefitting from Title ll-C funds. In addition, Missouri has 
created standards and measures for other programs funded with Perkins funds (i.e., 
supplemental vocational education programs, community based programs, and 
programs serving single parents, displaced homemakers, and single pregnant women). 

Measures and Standards and Special Populations: 

The secondary and postsecondary preparatory program measures are: 

academic attainment, 

academic gain, 

occupational competence, 

student access (special populations), and 

placement. 

The attainment of basic and advanced academic skills is demonstrated through 
mastery of 70% of the key skills (in mathematics, English and/or communications, 
science, and social studies and/or science) on the 10th grade Missouri Mastery and 
Achievement Test during the 12th grade or, at the postsecondary level, mastery of 70% 
of the key skills on an institutionally-selected, Missouri Education Department-approved 
test at the completion of the vocational program of study. Academic gain at the 
secondary level is measured by the amount of change in normal curve equivalent 
(NCE) scores on the above tests between the 10th and 12th grade testing. The 
standard for change is that the 12th grade NCE is equal to or greater than zero. 
Postsecondary academic gain is measured by calculating the change in test scores in a 
program entry - program completion testing sequence. 

Occupational competence is defined as mastery of 80% of the essential occupational 
and employability skills necessary for entry-level employment as identified by the 
institution and approved by the Missouri State Education Department. 

The 85% placement standard is inclusive of all further employment, education or 
training, and military service. The rate is established by the number of vocational 
education students who have been enrolled in an approved vocational education 
program or course. 
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Access is defined as the participation rate of individuals who are members of special 
population groups. Participation of these groups is to be equal to or greater than the 
proportion of their membership in the relevant district or institution population 

In addition, students whose education experience is modified by an Individualized 
Education Plan (IEP) are excluded in the calculations for measures of academic 
attainment, academic gain or occupational competence. 

Evaluation Process: 

Annually, each recipient reports the results of the local evaluation in the local 
application for the next fiscal year. While the local evaluation must be, at a minimum, 
based upon the state system of performance standards and measures, local recipients 
may expand their local evaluation process to fit their needs. 

Incentives, Adjustments, and Modifications: 

Adjustments are established for each measure to allow: 

exclusions from the calculation rates (i.e., transfer students without 
achievement test scores), or 

interim standards that ensure continuous improvement, (i.e., districts not 
achieving the access standard are to maintain a 5% annual gain), or 

a selection of calculation methods from which local recipients may choose 
(i.e., placement may be calculated by individual program, course, or 
aggregated to the institution level, and it may be calculated either annually 
or on a three year average). 
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MONTANA 



Type of System: 

Montana has developed a singular system of program performance standards and measures 
to evaluate all secondary and postsecondary vocational education programs in the state. 



Measures and Standards: 

The system includes the following measures. 

academic competency gain, 
competency attainment, 
retention, and 
placement. 

Progress in achievement of basic and more advanced academic skills is indicated when 80% 
of the students demonstrate an increase in score on a pre/post-test assessment. Local 
education agencies may use either a standardized instrument or a teacher or faculty 
developed, locally-referenced test. 



Competency attainment is measured by calculating the percentage of program completers. 

The standard is that a program demonstrate an increase in the percent of students completing 
a funded program. 

The retention standard is a documented increase in the school retention rate of vocational 
students by funded program compared against the previous three year average or by some 
other relevant baseline data. 

Placement is measured by calculating the percent of students placed into additional training or 
education, military service, or employment. 

Incentives, Adjustments, Modifications and Special Populations: 

Montana has incorporated both modification and incentive into the system Eligible recipients 
may explain why and suggest a more appropriate standard for academic gain if more than 
20% of the students served cannot be expected to make progress according to standardized 

or local tests. 

As an incentive to encourage service to special populations, Montana recognizes the ten 
programs that demonstrate the greatest increase in services to at least four of the five special 
population groups. 

Future Activities: 

Proposed future incentives include a monetary incentive to those top ten local districts 
demonstrating the greatest increase in services to special population groups. Implementation 
is pending approval to use up to 6% of the formula-allocated grant to fund this incentive. 
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NEBRASKA 



Type of System: 

Nebraska has a singular system of performance standards and measures to evaluate 
the secondary and postsecondary vocational education programs. 

Measures and Standards and Special Populations: 

The ten measures are: 

academic gain, 
occupational attainment, 
employability skills attainment, 
retention in school, 
student satisfaction, 
placement, 

placement (special populations), 
earnings, 

employer satisfaction and, 
participation (special populations). 

Academic gain is demonstrated by an increase in scores on a standardized test of 
academic gain; A minimum of 80% of vocational students will demonstrate gains equal 
to that of the total student population in that grade level. 

Occupational attainment measures a student's mastery of competencies as listed on an 
occupational competency exam appropriate for the vocational program area. A 
minimum of 80% of vocational students will demonstrate mastery of 80% of the 
appropriate competencies. Employability skills attainment refers to work entry 
competencies such as those identified in Workp l ace Read i n e s s or Youth works . A 
minimum of 80% of students will demonstrate mastery of 80% of the employability skills 
competencies by graduation. 

Retention in school is a comparison of the dropout rate of vocational students as 
compared with the dropout rate of the entire student population. The standard is that 
the dropout rate of vocational students will be less than that of the entire student 
population. 

Student satisfaction with the vocational and academic skills acquired in the educational 
program is measured using the ratings reported by vocational students on graduate 
surveys administered within one year of graduation. Using separate questions for 
rating academic and vocational skills, the standard for each is that 80% of vocational 
graduates responding to the survey will rate the skills attained as satisfactory or better 
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(3 or higher on a 4 point Likert scale) Employer satisfaction with the vocational 
program graduate employee’s vocational and academic skills is also polled using 
employer satisfaction surveys administered within one year of the student's 
employment. Again, using separate questions to rate academic and vocational skills, a 
minimum of 80% of vocational program completers employed in areas related to their 
training will receive satisfactory ratings by their employer (3 or higher on a 4 point 
scale). 

Placement measures the employment and education activities of vocational education 
program completers within six months of graduation. The standard is that 80% of 
vo ca tional program completers will attain placement in a job, in the military, or in a 
postsecondary education program within six months of graduation. In addition to the 
analysis of placement data for all vocational completers, data are disaggregated to 
monitor the placement rate of special population students against the same standard. 

Earnings measures the percentage of vocational program completers earning a higher 
entry level wage or receiving advanced placement (or hiring advantage) as compared 
to entry level workers who are not vocational education program completers. The 
standard is that the percentage of vocational education program completers receiving 
higher entry level wage or advanced placement (or hiring advantage) will be greater 
than for those who did not complete a vocational education program. 

Participation measures the enrollment rate of special population students in vocational 
education programs. The standard for the participation of special population students 
in vocational education programs is that it is at least the same as the participation rate 
of the entire school population. 

Evaluation Process: 

In Nebraska, responsibility for assessment and program improvement rests with the 
local institution. No data are collected to determine the extent to which local education 
agencies are implementing and meeting the performance standards. Technical 
assistance is available on request. 
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NEVADA 



Type of System: 

Nevada has developed a singular system of measures and standards to evaluate 
secondary and postsecondary and adult vocational education programs receiving Title II 
funding. While the types of measures are identical, the measurement indicator deinition 
is unique to the level of program. 

Measures and Standards: 

The measures are: 

achievement of basic/advanced academic skills, 
occupational competency gain, 
retention (secondary only), and 
placement. 

The achievement of basic academic skills is determined by successful passage of the 
Nevada High School Proficiency Exam (NHSPE). Passing the NHSPE is a basic 
requirement for earning a high school diploma. At least 92% of the occupational students 
who initially fail the exam in 1 1th grade will successfully pass the exam within the 
following year. The achievement of advanced academic skills is determined by the 
percentage of occupational program completers successfully earning at least one 
advanced science or mathematics credit by the end of their senior year. The state 
established standard is that at least 45% of the occupational program completers will 
earn at least one advanced mathematics or science credit. 

At the postsecondary and adult level, achievement of basic academic skills is measured 
by the percentage of occupational students successfully completing a basic academic 
skills course ( a pre-collegiate level general education remedial course) compared with 
the percentage of all full-time students completing a basic skills course. At least 65% of 
full-time occupational students will successfully complete such a course. 

Postsecondary programs measure the achievement of advanced academic skills by 
calculating the percentage of occupational students who successfully complete six credits 
of advanced English or communication courses as compared with the number of 
occupational education students who attempted such courses. Adult programs measure 
advanced academic achievement by the percentage of students in occupational programs 
who successfully complete the requirements for a high school diploma or GED compared 
with the percentage who are enrolled in an occupational program. For both 
postsecondary and adult programs the standard is that at least 85% of occupational 
education students will successfully complete such courses or requirements. 
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At the secondary level, occupational competency gain applies to those programs that 
provide business and industry validated occupational competency certificates to program 
completers. The measure is the number of occupational program completers who 
demonstrate mastery of at least 80% of the competencies identified on the certificate. 
Mastery is defined as performance at a level equal to or greater than business and 
industry requirements as determined by a technical skills committee. 

At the postsecondary level, occupational competency gain applies to those programs for 
which there are certification or licensure examinations administered by a third party 
agency or organization. The percentage of program completers successfully passing 
certification or licensure examinations compared with all test-takers will be within five 
percent of the state-wide or national percentage. 

Retention measures the dropout rate of students enrolled in occupational courses as 
compared with the dropout rate of the total district student population. The standard is 
that the percentage of students enrolled in occupational courses who dropout will be less 
than the percentage of students for the district as a whole. 

For all programs, placement measures the percentage of occupational program 
completers reported in continued education or training programs, employed in a job at 
least 20 hours per week, or entered in military or foreign service for approximately six 
months (onel year for postsecondary and adult programs) after program completion. The 
state-established standard is for at least 90% of secondary and adult occupational 
program completers and 92% of postsecondary program completers to be in one or more 
of the placement categories. 

Evaluation Process: 

Sub-recipients conduct the annual evaluation and forward results of the evaluation to the 
state. In addition to local compilation of data, state-wide dropout and proficiency exam 
data will be used in the reporting requirements. 

• 

Special Populations: 

Special population students are expected to adhere to the same measures and standards 
outlined for all program levels. 

Incentives, Adjustments, and Modifications: 

Local education agencies are to develop incentives and adjustments to the measures to 
provide for the full participation of special population students. Adjustments may include 
smaller class size, modifications to equipment, alternative assessment instruments, 
optional graduation diplomas, adjustments to time, and program requirements. Incentives 
may include additional staff development release time, special recognition for sites, 
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programs, teachers, and administrators who exceed state-wide standards and/or excel in 
working with special population occupational students. 



O 

ERIC 
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NEW HAMPSHIRE 



Type of System: 

New Hampshire has developed separate measures and standards to evaluate all the 
secondary and postsecondary vocational education programs of recipients of Perkins 
funds. 

Measures and Standards and Special Populations: 

The seven secondary measures are: 

academic gain, 
occupational competency, 
program completion, 
school-to-work transition, 
enrollment (special population), 
enrollment (non-traditional), and 
student satisfaction. 

Recently revised, the academic gain measure now requires measurement using a pre- 
post-test methodology. Choice of assessment instrument is a local decision. However, 
the instrument must be capable of assessing both basic and advanced academic skills 
in mathematics and language arts. It must also be (1) appropriate to the student 
population served, (2) sensitive enough to measure gain, and (3) reliable and valid. 

The measure of academic gain has both a state and a local standard The state has 
established a standard that 100% of program completers will demonstrate academic 
gain commensurate with program objectives. In addition, the state has directed the 
local recipient to establish a specific amount of academic gain to be achieved in each 
program. 

Occupational competency is measured by attainment of core competencies established 
for the program. The standard is that not less than 80% of program completers will 
attain 100% of the core competencies. The standard for the program completion 
measure is that not less than 80% of students enrolled in a program will complete it. 

The school-to-work transition measure is determined from the results of a follow-on 
survey of program leavers and completers eight months after program completion. The 
state-established standard is that not less than 80% of program completers responding 
are either employed and/or continuing their education. 
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Program enrollment patterns are measured to identify the distribution of special 
populations across all programs and the proportion of non-traditional enrollments within 
programs. The standards are, respectively, demonstration of a movement toward 

equalizing the distribution of special populations from one year to the next, and an 
increase in the proportion of non-traditional enrollments from one year to the next. 

Student satisfaction with the quality of their vocational program is captured on the 
follow-on survey. Rating the quality of the vocational program on a scale of 1-4, the 
standard to be attained for each program is an average score of 2.0 or greater ("Good" 
to "Very Good") by 85% of the students surveyed. 

The postsecondary system contains two measures: 

academic achievement and 
retention. 

The indicator used to determine academic achievement is the cumulative grade point 
average of program participants. The state standard is that 70% of students enrolled 
in the program will achieve a cumulative grade point average of 2.0 or higher. 

Retention is determined by calculating the percentage of students enrolled in the 
program at the beginning of the fall semester who are still enrolled at the end of that 
semester. The standard is 60%. 

Evaluation Process: 

Local recipients include the results of their evaluation of basic and advanced academic 
gain in their next year's application for funding. In the application, the local recipient 
identifies (1) the assessment instrument(s) used, (2) the local standard to be 
achieved, and (3) the method used in reporting data on aggregate student gains by 
program and special population group. 

To evaluate and report on achievement of the school-to-work transition measure, 
recipients can choose to conduct either a survey of all students (25+% response rate 
required) or may survey a random sample of 50% of the student population (50+% 
response rate required). The survey instruments are provided by the state. Local 
recipients distribute, collect, and process the surveys and provide the data to the State 
Department of Education on magnetic tape or floppy disc. The state then provides 
each local with a summary report of its results including breakouts by program and 
special population group. 

The state collects and compiles data on achievement of each of the measures by 
institution at the secondary and postsecondary level. 
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Incentives, Adjustments, end Modificetions: 

Adjustments are provided in the calculation of academic gain and occupational 
competency. In those instances where the assessment instrument used to measure 
academic gain is inappropriate for a student with special needs, an alternative form of 
assessment may be used. In such cases, the gain measures for these students are 
reported separately. 

Local recipients may modify the program or institutional standards based upon analysis 
of the previous three years' data or if evidence is provided to substantiate that the 
previous three years' data do not adequately reflect current conditions. However, 
modified standards must demonstrate a goal of movement toward improving the 
program outcomes of the previous years. 



NEW JERSEY 



Type of System: 

New Jersey has established a singular system of performance standards and 
measures to evaluate both secondary and postsecondary programs. 

Measures and Standards and Special Populations: 

The state’s system includes 17 indicators which are outlined in the New Jerse y 
Program Performanra Measures and St a n da rds for Occupat i ona l and Techn i c al 
Programs These include: 

basic and advanced academic skill gain and achievement, 

marketable occupational competency gain and achievement, 

employer and learner satisfaction with employment preparation, 

job placement, 

job entrance wage, 

program retention or completion, and 

the participation of members of special populations (recruitment, retention and 
program placement, further education or training interest, learner satisfaction, 
and enrollment in further education or training). 

Evaluation Process: 

These indicators are implemented through an Occupational Competencies Project, an 
Assessment Testing Process, and a Job Placement Accountability System. 

The Occupational Competencies Project has identified 101 occupations with the 
highest anticipated annual job openings. During 1992-94, the critical competencies 
were identified and verified utilizing a system patterned after the DACUM process used 
by V-Tecs. This list, which includes both academic skills as well as occupational 
competencies, is the basis for the measurement of performance. With the assistance 
of the Northeast Curriculum Coordination Center, tests have been identified or 
developed to assess the academic and occupational competencies. 

State-initiated technical assistance activities are being developed and assistance is 
being provided to local districts upon request. 
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NEW MEXICO 



Type of System: 

New Mexico has implemented a singular system of performance standards and 
measures to evaluate all secondary and postsecondary vocational education programs. 

Measures and Standards and Special Populations: 

The system includes measures in three areas: 

student gains, 
placement, and 
accessibility. 

Two measures of student gain are implemented at the secondary level: gain in both 
basic math and English and gain in vocational course completion. Academic gain in 
basic skills is measured through the use of the Iowa Test of Basic Skills (ITBS) during 
the eighth grade to establish baselines and administration of the New Mexico 
Competency Exam (NMCE) in the 10th grade. Advanced academic and vocational skill 
attainment is determined by the rate of completion of courses, programs, and 
competencies. Successful completion is defined as completion with a grade of "C" or 
fulfillment of the course requirements in a satisfactory manner. 

At the postsecondary level, basic academic gain is based upon obtaining a threshold 
score on a standardized test (i.e. , ACT Enhanced, TABE, GED, CPT, or ASSET) or 
successful completion of a developmental course. Advanced academic, vocational, 
and occupational skill attainment are measured by the number of students receiving 
degrees and certificates and the passage rates for national and state certification and 
licensing. In 1 994, the state set standards for student gain at both the secondary and 
postsecondary level. 

Placement at both the secondary and postsecondary levels is determined by the 
number of program completers who are employed in a job related to their training 
program, unrelated to their training program, enrolled in further education, or enlisted 
into military service. Standards will be established for placement in related and 
unrelated employment. 

Accessibility is measured by comparing rates of enrollment of special population groups 
in each vocational program with their representation in the school district (secondary) or 
the service area (postsecondary). The standard for both levels is that special 
populations be represented in vocational programs at a threshold level of 80% of their 
representation in the school district or service area. To reinforce sex equity, 25% is the 
pre-established threshold level for male/female enrollment in nontraditional programs. 
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Evaluation Process: 



Numerous state and federal, education and labor data bases (i.e., census data, 
unemployment insurance wage rate, military induction records) provide information on 
the measures. All data flows into the State Department of Education to be processed 
into a profile for each school district and postsecondary institution. Once the profiles 
are compiled, they are forwarded to the district or the institution to be used in program 
improvement and planning. 

Incentives, Adjustments, and Modifications: 

Schools meeting- or exceeding all standards will receive recognition as Outstanding 
Vocational Programs. This listing will be presented to the State Board of Education, 
listed in the State Vocational Newsletter, and schools will receive a certificate of 
outstanding performance. 

Future Activities: 

Additional incentives currently being explored include extending the time between state 
monitoring visits, canceling monitoring of selected local agencies, or permitting selected 
local recipients to omit specified items from their funding proposals. 
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NEW YORK 



Type of System: 

New York has established performance standards and measures for all secondary, 
postsecondary and adult vocational education programs in institutions receiving Perkins 
funds. 

Measures and Standards and Special Populations: 

Four measures and standards were adopted for secondary programs: 

basic academic skills attainment, 
program completion, 

program completion (special population), and 
program success (placement). 

Academic attainment is measured by calculating the percentage of program completers 
(those who complete an occupational sequence of three or more units) successfully 
passing the Regents Competency Tests (RCT) in mathematics, reading, and writing. 
The state standard is 100%. The State is piloting an expanded system of standards for 
occupational education that consists of advanced academic and personal work skills. 

Two standards were established for program completion, a basic standard of 76% and 
an advanced standard of 95%. Program success is measured by the percentage of 
program completers who are employed or pursuing additional education. Two 
standards have been established, a basic standard of 77% and an advanced standard 
of 98%. 

Separate measures of program completion and program success are established for 
disadvantaged and disabled students. Agencies are expected to close, by two percent 
per year, the gap between the rate for special population groups and the total 
occupational education population. 

The measures in the postsecondary and adult systems are. 

program completion, 
employment or continuing education, 
targeted enrollment, 
targeted success, and 
skill acquisition. 
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The measures of placement, employment and targeted success are applied only to 
credit and non-credit programs intended to lead to employment Programs leading 
simply to skills enhancement are not being measured or assessed The enrollment 

(participation) and success (placement) of special populations are evaluated, as they 
are in the secondary system. 

Evaluation Process: 

At the secondary level, first year evaluation data have been collected and submitted to 
the state for review. The results of two years' evaluations are being used by local 
agencies which are conducting a comparative analysis of the results for progress 
towards attainment of the standards. Local improvement plans will be developed as 
appropriate and implemented during 1995-96. The state will monitor the 1995-96 local 
evaluation. 

At the postsecondary level, methodology for the measurement of all indicators is in 
place with the exception of skill acquisition. Data collection and analysis are 
aggregated for credit versus non-credit student and/or program status. Placement 
information is collected through the use of a graduate follow-up survey as no wage- 
reporting system exists. 

Future Activities: 

At the secondary level, the state is assessing whether to develop a measure of job 
competency as part of the system of standards and measures. This will be predicated 
on the progress of the federal skill standards projects, the work of the State Curriculum 
and Assessment Committee to develop job competency assessments, and the 
establishment and direction of Career Major Panels established as part of the state's 
School-To-Work Opportunities Act planning. In addition, the state is revisiting how 
academic attainment is currently measured as the Regents Competency Tests will no 
longer be used upon the development of the Assessment of Initial Mastery. 

During 1993, at the postsecondary level, the state conducted a Student Skill Acquisition 
Survey of all programs to assess current practices for identifying, measuring, and 
documenting skill acquisition. At the conclusion of the initial survey, and as a next step 
in the process of establishing a measure of skill acquisition, institutions will be asked to 
submit institution-wide student skill acquisition assessment plans. 

To evaluate issues such as long-term placement, employer perceptions and student 
satisfaction, targeted follow-ups (of a state-wide sample of program completers) will be 
undertaken for both credit and non-credit programs focusing on specific occupational 
areas. 
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Standards for postsecondary and adult measures will be established through a public 
forum after review of the 1992-93 evaluation data. A process for the periodic review of 
the standards is planned for development. 



NORTH CAROLINA 



Type of System: 

North Carolina has established separate systems of performance standards and measures 
to evaluate all vocational education programs in the state. 



Measures and Standards and Special Populations. 
The eight secondary measures are. 



career development, 
enrollment (special populations), 
competency achievement, 
competency gain (two measures), 
placement (education), 
employment, and 
employment (related). 



Career development is measured by the number of students having a career development 
plan (CDP) on file which includes academic and vocational courses appropnate for the 
student's designated career goal. The standard is for every student to have a 

Enrollment measures the participation of special population students in e a ch vocational 
education program. The standard is that enrollments of special population students in each 
program area will not differ more than a third from the overall special population 
percentage enrolled in all vocational education programs in the school. Deviations must be 
supported by documentation of student choices as reflected in student s CDP s. 



Competency achievement measures the percentage of students mastering 80 /o of the core 
competencies in each course (as designated by the state-wide course bluepnnt). The 
standard is that 80% of the students will master 80% of the competencies. 



Competency gain is measured on both Level I, non-sequential vocational courses, and 
Level II courses. Eighty percent of all students completing a Level I course will ha ^ e 
named a minimum of 60% of the difference between the pre-test score and the post-test 
score as measured by tests of the core competencies designated on the state-wide course 
blueprint Eighty percent of the students completing a Level II course will have gained a 
minimum of 40% of the difference between a pre-test and post-test score as measured by 
tests of the core competencies as designated on the state-wide course bluepnn 



Placement is a measure of the number of students entering further education or training 
including that received in military service and on-the-job training, The standard is that 70 /o 
of all vocational education completers will enter further education. 



Two employment rates are calculated by program area for vocational program completers. 
The unemployment rate for completers seeking full-time employment is calculated It will 
be lower than the county's youth unemployment rate. In addition, of completers finding full- 
time employment, 70% will be employed in jobs related to their vocational program. 

The postsecondary measures are: 

learning and competency gain, 
retention, and 

participation (special populations). 

Learning and competency gain are measured by calculating the percentage of students 
completing 25%, 50%, 75%, and 100% of the courses needed for curriculum completion or 
graduation. Also reported are the rates at which vocational education students are 
required to take and pass remedial basic academic courses and general education and 
related courses. 



Retention rates are reported by credit hour categories (<5, 6-8, 9-11, and >12 credit hours). 
Retention rates are calculated by determining the number of students enrolled in the fall 
quarter who did not graduate that quarter and subsequently enrolled and completed at 
least one additional course during the following winter or spring quarter. Students enrolled 
in enrichment or enhancement courses, special credit, or dual enrollment or Huskins Bill 
courses are omitted from the cohort. 

Postsecondary institutions may choose from two methods to measure the participation of 
special population students in vocational education. The ratio of participation may be 
calculated on the percent of special population students enrolled in vocational curricula to 
the percent of special population students enrolled in all curricula programs in the school. 
Method two is to calculate the percent of special population students completing vocational 
curricula to the percent of special population students completing all curricula programs in 
the school population. 

No standards are established. 

Evaluation Process: 

At the secondary level, local educational agencies describe in their annual application the 
strategies used to improve the access, progress, and success in attaining the performance 
standards. The State Board publishes overall statistics at the local education agency level 
and at the program area level in each school for five of the eight performance standards. 
The local agency uses this information to establish benchmarks. The Board than publishes 
statistics to indicate progress in meeting these standards. The information is presented for 
both regular and special populations students. 

At the postsecondary level a centralized data collection system is being developed. 
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NORTH DAKOTA 



Type of System: 

North Dakota has established a singular system of program performance measures and 
standards to evaluate all secondary and postsecondary vocational programs. Performance 
standards are the second component of the state’s three-component, Standards of Qua li t y 
evaluation and continuous improvement system (quality of design and redesign, quality o 
performance, quality of conformance). 

Measures and Standards: 

The performance measures are: 

learner outcomes, and 
education/employment outcomes. 

Learner outcomes refer to student progress in attainment of academic skills and 
occupational-technical skills. Two sub-measures address learner outcomes. At the 
secondary level, progress is measured by comparing the 8th grade and 12t ^ 9 rade scores 
of vocational students on the state-wide Comprehensive Test of Basic Skills (CTBS). The 
standard for this measure is a minimum gain of .1 normal curve equivalent. In both 
secondary and postsecondary programs. 80% of program completers must demonstrate 
attainment of 80% of the program competencies. 

Education/employment outcome refers to the post-program placement of occupational 
program completers. A positive placement is defined as military service, employment 
related to the training program, apprenticeship, additional high school or college education 
(secondary) or additional college education in a related field or other vocational technical 
education program (postsecondary). The program standard is a 75% positive placement 
based on a 6-month follow-up of program completers from the previous year. No more 
than 10% of program completers may be reported as status unknown. 

Incentives, Adjustments and Modifications: 

Student attainment of 80% of the competencies and/or tasks identified for a program may 
be modified through a student's Individual Vocational Education Plan. 

Evaluation Process: 

Local education agencies annually complete the three-component Standards of Qual i ty 
document as a self-evaluation. Results of this self-evaluation are forwarded to the state. 
The State Board reviews each self evaluation for effectiveness. Local education agencies 
ultimately receive a letter identifying areas of non-compliance. Technical assistance is 
provided on an as-needed basis. 
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OHIO 



Type of System: 

Ohio has developed three sets of core measures and performance standards to 
evaluate all secondary, full-time adult, and postsecondary (associate degree) 
vocational programs. The Ohio Board of Regents has requested, however, that the 
postsecondary standards be applied to only those postsecondary programs receiving 
federal funds. 

Measures and Standards and Special Populations: 

The measures forming the secondary and full-time adult systems include: 

awareness of vocational education opportunities, 

basic, advanced, and applied academic learning, 

occupational competency, 

placement, 

enrollment, 

retention, 

career development (secondary only), and 
work and family life competency (secondary only). 

The awareness measure includes informing students and parents of general career and 
specific course information as well as information on eligibility, application procedures, 
and available support services. 



In the secondary system, passage of the ninth grade proficiency test, which is required 
prior to award of the high school diploma, is used to measure basic and advanced 
academic learning. Applied academics and occupational competency are measured 
using the vocational education competency tests offered by the Instructional Materials 
Laboratory at Ohio State University or an alternative standardized test. 

Multiple indicators (primary and secondary) have been established for measuring 
placement in an occupation or in further education. Satisfactory rates of placement are 
higher for program completers at the postsecondary level than for program completers 
of secondary programs. For example, the percentage rates of employment in 
occupations related to training are 60% at the secondary level and 80% for full-time 
adult programs. 

The enrollment measure for secondary programs assesses the participation of special 
population group members in various vocational education programs, Occupational 
Work Experience Programs, Occupational Work Adjustment Programs, and Work and 
Family Life Programs. The established standard is a threshold of 25% for "non- 
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traditional" enrollment in secondary vocational programs. Likewise, the enrollment 
measure for full-time adult programs assesses the participation rates of special 
populations in full-time vocational programs and the gender demographics of 
applicants. 

The secondary system also measures the learning and competency gains of students 
(special population) enrolled in Work and Family Programs, including those enrolled in 
GRADS, using vocational education competency tests offered by the Vocational 
Instructional Materials Laboratory at The Ohio State University. 



The retention measure within the secondary system addresses the dropout rate of 
GRADS students and establishes a rate no higher than 15%. For programs serving 
adults, retention is measured on positive program outcomes: program completion, 
transfer to further education, or employment related to the occupational area. The full- 
time adult program retention standard is 65%. 



The secondary system also includes an additional measure: career development. 
Career development is measured by identifying the percentage (75%) of students 
articulating an occupational goal and an educational plan to achieve that goal. This 
data will be captured through random sample interviews. 

Core standards and performance measures for postsecondary associate degree 
institutions build on already successful practices established by each institution. The 
measures include: 



basic academic skills, 
advanced academic skills, 
placement, 

occupational competency (proposed), and 
enrollment/graduation of special populations (proposed). 

All students are tested upon enrollment to identify any need for remedial instruction. 
Those students lacking the necessary competencies for the degree program are 
enrolled in developmental courses. The successful completion of developmental 
courses is the measurement used to determine gain in basic academic skills. 
Advanced academic learning gain is measured by successful completion of general 
education courses (English and/or communications) required in the associate degree 
program. 

As economic conditions vary across Ohio, local variance to the placement standard 
may be justified. However, the state-established standard is 75% in training-related 
employment, 90% total employment, and 95% employed or enrolled in additional 
training, military or foreign aid service. 
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Implementation of measures of occupational competency and the 
enrollment/graduation of special populations is under consideration and data are being 
collected and evaluated. 

Incentives, Adjustments, and Modifications: 

Ohio has established a set of both positive and negative incentives. Positive incentives 
include: higher priority for receipt of sex equity grants, reduced reporting requirements, 
and recognition awards for service to special populations. Negative incentives include: 
withdrawal of units that fall below enrollment/placement standards, lower priority for 
competitive grants, increased reporting requirements, and reduced priority for receipt of 
new secondary vocational education units. 



OKLAHOMA 



Type of System: 

Oklahoma has developed a singular system of program performance measures and 
standards to evaluate all vocational programs. 

Measures and Standards: 

The system includes four measures. They are: 

learning and competency gain, 

competency attainment including work skill attainment, 
retention in school or completion of secondary school, and 
placement. 

Learning and competency gains are measured by determining the modified percent 
gain score computed from raw score data gathered from administration of a series of 
matched tests throughout the duration of each vocational program (a minimum of four 
testings per year). The standard is that each student will demonstrate a modified 
percent gain of 20%, 40%, and 60% respectively in FY93, FY94, and FY95 on both 
total test score and the academic skill component. Gains testing and relevant 
recordkeeping is conducted entirely at the local level. 

Competency attainment is determined through the administration of competency tests 
(based upon occupational duty or task lists) which include both written and 
performance components. The standard is that 1 00% of vocational students who 
complete occupational training will be competent in a least one occupation. 
Competency in a specific occupation is achieved when a student scores a minimum of 
70% on the cognitive assessment and 100% on the performance assessment. 
Assessments are developed and scored at the state level and administered at the local 

level. 

Retention and/or completion are measured by comparing the number of vocational 
completers with the total number of secondary school graduates. The standard is that 
at least 90% of students who complete an approved vocational program will complete 
secondary school or its equivalent. 

Placement measures the percentage of program completers placed in additional 
education, or employment including military service. For this measure, program 
completers are defined as any student who has finished a planned sequence of 
instruction designed to meet an occupational objective as identified by their instructor. 
The student may or may not have completed the entire vocational program curriculum 
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content. The placement standard, based on the known status of all program 
completers, is a rate of 90%. 

Evaluation Process: 

The evaluation process is a joint state and local effort. The state provides leadership to 
locals through the development of evaluation procedures, occupational competency 
assessments and scoring of competency tests, system implementation guides, 
computer software, and training sessions. Evaluating the progress of local programs in 
implementing and meeting the state standards is a part of the monitoring and technical 
assistance process. The state monitors the progress of local programs in implementing 
and meeting the standards through site monitoring visits and through analysis of the 
competency assessments forwarded to the state for scoring. Gains testing and all 
relevant recordkeeping are conducted entirely at the local level. If a program does not 
meet a standard, the program instructor develops a program improvement plan. Also, 
each instructor is required to complete and return a report identifying specific progress 
for each standard to the monitoring staff at the State Department of Vo-Tech. 

Incentives, Adjustments end Modifications: 

Locals may make local modifications to the system of measures and standards based 
on conditions and factors identified in the Act as long as the program serves a 
designated level of students meeting these conditions or factors. 

A procedure is established to provide adjustments to the testing situation for programs 
serving target groups. Some adjustments which may be made to the testing situation 
include testing time limits, oral administration, test length, readability, etc. 



Future Activities: 

The state is continuing development of additional duty or task lists and assessment 
batteries while continuing to update and validate existing task lists and test item banks. 



O 

ERIC 
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OREGON 



Type of System: 

Oregon has developed a singular system of performance standards and measures for 
evaluating secondary state-approved, vocational education programs within each of the 
six Certificate of Advanced Mastery areas and postsecondary vocational programs 
leading toward an associate degree, diploma, or certificate. 

Measures and Standards and Special Populations: 

The system includes four measures. They are: 

enrollment, 
satisfactory progress, 
program completion, and 
placement. 

Each of the four measures includes an auxiliary measure calculating the rate of 
achievement of special population and protected class students. The standards for the 
auxiliary measures are the same as for all students. 

Enrollment is a measure of the rate at which students enroll in professional and 
technical programs. The standard is 40% and 50% enrollment rates by 1995 and 1996 
respectively. 

Satisfactory progress is the timely and efficient attainment of all requirements of a 
performance-based professional, technical education program area. During the 
transition from traditional to performance-based learning, satisfactory progress is 
defined as attainment of a 2.0 cumulative grade point average during the enrollment 
year. The standard for satisfactory progress is 1 00 percent by 1 995 and each year 
thereafter. 

Program completion is a measure of the rate at which students in professional technical 
education programs are awarded Certificates of Advanced Mastery, high school or 
community college diplomas, associate degrees, or have attained two thirds of all 
outcomes of a performance-based program. This is applicable to students enrolled in 
programs in which they have declared a major or declared an intent to complete all 
requirements for a degree. Students may not have already attained a degree or 
certificate in the occupational program area or not re-enrolled for three consecutive 
terms. The standard is that the rate at which professional technical education students 
become program completers is 83% by 1995 and 85% by 1996. The standard for 
community college program completers is 60% by 1995 and again in 1996. 
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Placement is measured by calculating the rate at which professional technical program 
completers are employed or pursue additional education or training in fields related to 
their professional technical preparation. Until a reliable method for capturing data on 
further education is established, the interim measure and standard is that 70% (1995) 
and 75% (1996) of professional technical education program completers are employed. 

Evaluation Process: 

The evaluation system incorporates both process and outcome standards to support 
Oregon's Educational Act for the 21st Century. Outcome standards (standards and 
measures) are reported annually on every state-approved professional technical 
program area within each of the six Certificate of Advanced Mastery (CAM) areas. 
Responsibility and accountability are delegated to the local level while the state s role is 
focused on providing training and technical assistance. 



The evaluation system provides for all data reporting through the Office of Professional 
Technical Education Management Information System for secondary programs and the 
Oregon Community College Unified Reporting System for all postsecondary programs. 
The system provides for linkage to the Department of Employment's unemployment 
insurance employer wage records to obtain follow-up data through the Oregon 
Automated Follow-up System. 

Incentives, Adjustments, and Modifications: 

Local education agencies may modify measures and standards based upon the 
conditions and factors described in the Act. 
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PENNSYLVANIA 



Type of System: 

Pennsylvania has developed separate systems of performance standards and 
measures to evaluate secondary and postsecondary vocational programs. At the 
secondary level, annual evaluations apply to all state-approved programs. At the 
postsecondary level, all occupational specific programs (including adult) at institutions 
receiving Perkins funds are included in the annual evaluation requirement. 

Measures and Standards: 

The secondary system contains three measures. They are. 

basic and advanced academic skills, 
occupational competency, and 
placement. 

Gain in basic academic skill is measured using the Pennsylvania System of School 
Assessment (reading and math). The established standard is that the average score of 
grade 12 vocational-technical students will be equal to or higher than the school 
average of all grade 1 1 students. Assessment of advanced academic skill is measured 
by the number of advanced academic courses (math, science, and communications, 
and including applied academics) completed by secondary vocational students. The 
standard established is that 35% of vocational completers will have completed six or 
more semesters of advanced academic courses. 

As of March 1 992, vocational-technical education programs in Pennsylvania are 
required to include content based upon occupational analysis, performance objectives 
deemed critical to success in employment, and student assessment based upon 
performance standards. Consequently, occupational competency gain is measured by 
the number of program completers. The standard is established at 70%. 

Placement rate is determined by counting the number of program completers who are 
employed, in further education, or in military service. A satisfactory placement rate is 

90%. 

The postsecondary system measures are: 

basic and advanced skills proficiency, 
occupational competency, 
program completion, and 
placement. 
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Proficiency in basic and advanced academic skills and in occupational competency are 
measured using student grade point averages (GPA). Seventy percent of occupational 
students are to achieve a 2.0 GPA in their academic courses in a designated year to 
demonstrate academic proficiency and a 2.5 GPA in their occupational courses to 
demonstrate occupational competency. 

Program completion is based on the ratio of completers to enrollees. A completion rate 
not more than 8% below the aggregate educational institution completer rate is the 
established standard. 

Placement is determined by counting the number of program completers who are 
employed in occupations related to their training, in further education, or in military 
service. A satisfactory placement rate is 80%. 

Special Populations: 

The Pennsylvania Department of Education will use the results of the Office of Civil 
Rights reviews to determine the levels of access that special populations have to 
secondary and postsecondary vocational programs. Measuring the progress of special 
population students in meeting the standards is conducted locally and the results are 
included in each district's or institution's two-year plan. 

Evaluation Process: 

The first local evaluations were conducted in 1992-93. The state-developed a guide 
entitled Managing Local Plans to assist local agencies in understanding the importance 
of performance standards and measures and in conducting their evaluation. Local 
agencies are to use these results in developing their two-year plan for 1994-96. 

State-wide data were collected and compiled by the state in the spring/summer 1993. 
The state will analyze the data and disseminate a report in 1994. Data for all measures 
with the exception of basic academic skills is available through the Vocational 
Education Management Information System. Data on academic skills will be collected 
through the Department’s School Assessment System. 

Incentives, Adjustments, and Modifications: 

In both the secondary and the postsecondary system, alternative standards (gain 
standards) which reflect progress towards achievement of the absolute standard have 
been established for each measure. Local modifications are allowable. Any 
modifications of these measures and standards by districts or institutions are described 
in the local application. 
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PUERTO RICO 



Type of System: 

Puerto Rico has developed a singular system of performance standards and measures 
to evaluate all vocational education programs receiving funds. 

Measures and Standards: 

The system includes five measures: 

academic achievement, 
occupational achievement, 
retention, 

access to special populations, and 
employability. 

Evaluation Process: 

The Information Component of the PR Vocational Planning Office, jointly with the 
Statistical Division of the PR Department of Education, have identified and validated 
instruments to be used for island-wide data collection. A study is underway to 
determine the feasibility of automating data collection. 

The Evaluation Component of the PR Vocational Planning Office is expanding the 
development of computerized item banks to measure competency attainment. 

Incentives, Adjustments, and Modifications: 

Sub-recipients are permitted to make local modifications to the system of core 
standards and measures when it can be demonstrated that conditions or factors 
mitigate against the achievement of the standards. Local proposals must clearly 
identify the differential factors and reflect that the proposed, modified standards and 
measures fully conform to the requirements of the Act. 
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RHODE ISLAND 



Type of System: 

Rhode Island has developed separate systems of performance standards and 
measures to evaluate regional planning district vocational-technical education 
programs. The measures and standards in both systems are organized into three 
categories - participation, achievement and customer satisfaction. 



Measures and Standards: 

The measures in both of the systems include: 

awareness of vocational education opportunities, 
attendance, 

occupational program completion, 
retention in career/technical center, 
accessibility to special populations, 
occupational competency, 
academic learning gains, 
graduation, 
placement, 

student satisfaction, and 
employer satisfaction. 

The awareness measure includes informing students and parents of general career and 
specific course information as well as information on eligibility and application 
procedures and support services available. In addition to measuring the enrollment of 
various special population groups in vocational programs, the measure of accessibility 
also includes a standard of 25% for "non-traditional" enrollment in secondary programs. 

Occupational competency is measured against competencies developed by state-wide 
craft committees or by scoring 50% on a norm-referenced standardized competency 
examination. The postsecondary level may also demonstrate occupational competency 
through licensure, employer satisfaction or articulation with advanced standing to a 
vocational program leading to a degree. 

Separate standards are identified for measures of basic and advanced academic skills. 
In addition to the standardized Metropolitan Achievement Test, skills may be 
demonstrated through projects, completion of advanced math or science courses, 
documented career planning, and employer feedback. Individual students scoring at or 
below 39% (36% for those identified as Limited English Proficient) are to receive 
supplemental instruction. 
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Multiple standards have been established for placement in further education or an 
occupation. Percentage rates for various sub-categories of educational or occupational 
placement and for various salary and wage rates have been established. The 
secondary system calls for both a six month and a long range follow-up. The 
postsecondary placement standard is placement within three months or the average 
placement time for the region. 
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SOUTH CAROLINA 



Type of System: 

South Carolina has developed separate systems of measures and standards to 
evaluate vocational education programs. Performance standards and measures apply 
to all secondary programs, adult programs funded with Title-ll funds, and 
postsecondary diploma/degree programs having comparable secondary-level courses 
and competencies articulated from the secondary level (or articulation agreements). 

Measures and Standards: 

The secondary measures are. 

academic gain, 
competency attainment, 
completion, and 
placement. 

At the secondary level, academic gain is demonstrated by the percent gain in scaled 
scores of program completers between grades nine and eleven on the Stanford-8 3R s 

Battery” test. 

Each local education agency may select one or more methodologies to measure 
competency attainment (i.e., occupational competency tests, profiles, classroom 
grades). The assessment selected must be designed to show gains in occupational 
competencies and is to be applied consistently over time. 

Completion is indicated by the number and percent of occupational program completers 
obtaining a high school diploma or certificate. 

Placement is indicated by the number of occupational program completers, other than 
agriculture education completers, employed in a position related to their area of 
training. The average of the prior three years is calculated to determine that an agency 
places at least 50% of its completers. 

The adult and postsecondary measures are: 

academic gain, 
competency attainment, 
retention (postsecondary only), and 
placement (postsecondary only). 
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Adult education programs may select one or more traditional assessment techniques 
(i.e., pre/post test, grades,) or performance assessments (i.e., South Carolina 
Occupational Competency Profiles, product analysis) to measure academic gain and 
competency attainment. 

To measure academic gain, postsecondary programs must demonstrate yearly gain in 
the number of tech prep students enrolled and achieving success in technical college 
courses and, by 1994-95, a decrease in the percentage of students requiring 
developmental/remedial courses. Each institution is to develop its own tracking system 
to monitor enrollments, successful program completion, and skills progress in college 
courses, including developmental programs. 

Likewise, postsecondary institutions are to develop an assessment system to measure 
competency attainment. Measures may include grades, grade point averages, 
completion, advanced credits, exemption from developmental courses, or the 
relationship of secondary coursework to entry-level postsecondary courses. 

Retention is determined by the number and percentages of program completers as 
compared to 1993-94 baseline data. Tech prep students will show positive and 
improving retention rates when compared to the general college rate during 1993-97. 

Placement refers to placement in a job, enrollment in higher education programs, or 
enlistment in the military service at rates equal to or above college-wide rates. This will 
be measured by the percentage of completers available for placement during and after 

1994-95. 

Absolute standards are not pre-determ ined nor established at the state level. Local 
districts and postsecondary institutions are to set their own projections of gain based 
upon past performance levels. 

Special Populations: 

No special populations or target groups may be excluded from inclusion in the 
measurement and calculation of any measure or standard. 

Evaluation f y rocess: 

Annual evaluations are a local responsibility. Local districts forwarded the results of the 
first annual evaluation using the measures and standards to the state in the annual 
Occupational Education Progress Report. This information was analyzed to determine 
the baseline for current and future progress. Postsecondary data for 1992-93 will not 
be available until Fall 1994. 



In addition to this report, the State Department of Education annually conducts on-site 
program evaluations of 20% of the local districts and postsecondary institutions to 
monitor and validate the implementation of the state-wide system. Also, the local 
application for funding must contain a written plan outlining the procedures for 
conducting the local evaluation for each measure, a narrative describing the incentives 
and adjustments used and their impact, evidence of consideration of the activities of 
other federal programs, and a local improvement plan, if applicable. 

Incentives , Adjustments, and Modifications: 

Responsibility is delegated to each local district and postsecondary institution to provide 
incentives or adjustments to encourage service to special populations. 



SOUTH DAKOTA 



Type of System: 

South Dakota has established a singular system of performance measures and 
standards to evaluate all vocational education programs. 

Measures and Standards: 

The six measures are: 

awareness of vocational education, 

program enrollment and completion (special populations), 

gain in basic/advanced academic skills, 

occupational competency gain, 

placement, and 

retention. 

Awareness of vocational education is measured by calculating the percentage of 
vocational educators making on-site business/industry visits, by calculating the number 
of programs conducting at least one public information event, and by calculating the 
percentage increase in program enrollment (postsecondary only). The standard is one 
on-site visit by all vocational education instructors, public events by all programs, and a 
5% increase in program enrollment. 

Program enrollment and completion by special population students is defined differently 
for secondary and postsecondary level programs. At the secondary level, it measures 
the percentage of special population students enrolled and completing programs. The 
percents are to be equal to the percents of non-special population group students in 
vocational education. At the postsecondary level, it is measured by the types of 
supplemental instruction and supplementary services provided to special population 
students and by the types of affirmative outreach and recruitment efforts used by the 
institution to recruit special population students. 

At the secondary level, gain in basic/advanced academic skills measures the 
percentage of student completers gaining a F or more normal curve equivalent on the 
Stanford Achievement Tests or Metropolitan Achievement Tests in reading, math and 
science between the 8th and 1 1th grades. At the postsecondary level, gain in basic 
and advanced academic skill is measured by identifying the number of students not 
possessing these skills at the beginning of the program with the number of completers 
who demonstrate gain in those skills associated with their program. 

Occupational competency gain measures student completers acquisition of 
occupationally specific skills. At the secondary level, 100% of program completers will 
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demonstrate attainment of 70% of the competencies associated with the program. At 
the postsecondary level, 100% of program completers will demonstrate attainment of 
80% of the competencies associated with the program and 80% of employers will 
indicate satisfaction with the occupational competencies of students. 

Placement measures students post-program activities. The standard is for 90% of 
secondary program completers to be placed into additional training, military, and/or 
employment while 75% of postsecondary program completers will gain employment or 
additional education in their identified program field. 

Retention measures the percentage of students enrolled in a vocational program who 
remain in school for an entire year. The standard is 90% at the secondary level and 
75% at the postsecondary level. 

Special Populations: 

Special populations are addressed through establishment of a secondary measure 
which monitors the enrollment and completion of special population students and which 
monitors supplemental instruction, support services and recruitment and outreach 
efforts for special populations at the postsecondary level. In addition, special 
population students may be considered to have achieved attainment of the 
occupational competency if they meet the course or program competencies established 
in their Individualized Vocational Education Program. Program calculations for 
determining attainment of the placement standard and the retention standard are to 
include students identified as traditionally gender-underrepresented and other members 
of special populations. 

Evaluation Process: 

Local evaluation forms, developed by the state, are used by local education programs 
in the annual evaluation process. These forms require local districts and institutions to 
provide an explanation of the activities and strategies to be implemented to meet the 
standard in the following year. Results are forwarded to the state. 

Incentives, Adjustments, and Modifications: 

The Committee of Practitioners hears recommendations for state-wide modification of 
the approved measures and standards at regularly scheduled meetings. The 
Committee requires written or oral testimony prior to any action regarding a system 
change. 
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TENNESSEE 



Type of System: 

Tennessee has developed separate systems of performance measures and standards to 
evaluate all secondary, area vocational-technical, and two-year institutions’ vocational 
programs. 

Measures and Standards: 

The secondary program measures are: 

academic gain, 
employer satisfaction, 
occupational attainment, and 
placement. 

Academic gain is measured using the WorkKeys assessment developed by American 
College Testing. Academic skills measured include applied mathematics, reading for 
information, listening, and writing. Aggregated test results will be used to establish 
baseline rates from which value-added standards will be established. 

Employer satisfaction measures employer satisfaction with the worksite performance of 
vocational completers in their employ. The standard is an average or above average rating 
from employers. 

Occupational attainment measures student mastery of occupationally specific 
competencies. The performance indicator is the percentage of students mastering 75% of 
the competencies in a competency based curriculum framework as determined by the 
classroom instructor. 

Placement is measured using a follow-on survey of students’ post-program employment 
and educational activities. The standard is that 80% of students will be placed in direct 
related employment by the year 2000. 

The measures for area vocational-technical school programs are. 

basic skills achievement, 

completion, 

placement, 

student satisfaction, and 
employer satisfaction. 

Basic skill achievement is a measure of the acquisition of academic competencies of full- 
time students relevant to their chosen occupational training program The standard is that 
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80% of students completing remedial classes will meet the academic competencies 
necessary for pursuing their chosen occupational program. 

School completion measures the rate of students completing programs with a diploma or 
competency certificate. The standard is that 40% of students eligible for termination will 
receive a diploma or competency certificate. 

Placement measures the number of program completers placed in employment. The 
standard is that 50% of program completers will be placed in employment. 

Measures of student and employer satisfaction calculate the percentage of program 
completers and employers of program completers who rate their satisfaction level with the 
level of training provided as either "good" or "excellent". The standard for both measures is 
a 70% satisfaction rate. 

The following measures are established for two year institutions: 

general education, 
major field, 
graduation, 
placement, and 
employer satisfaction. 

The indicator, general education, measures improvement in the quality of the associate 
degree general education program. This is accomplished by comparing the mean score of 
program completers on an approved standardized test of general education with either 
national norms or to previous performance. 

The measure "major field" addresses institutional improvement in the quality of the 
technical/career AAS program. A program will be considered successful if (1) the average 
score of graduates is either at or above a recognized norm or (2) the average score shows 
improvement over a previous testing. 

Graduation is measured using "persistence to graduation goals" developed by each 
institution. Persistence to graduation will be calculated by identifying all first-time, full-time 
enrollees in a specified semester and following the cohort through six years to establish 
graduation rates. 

Placement evaluates the post program activity of program graduates using a follow-on 
survey conducted 6-18 months after graduation. The established standard is a 75%. 

Employer satisfaction with the performance of program graduates is surveyed. Local 
institutions will establish appropriate positive response benchmarks of average or above 
average performance ratings by employers. 
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Evaluation Process and Incentives, Adjustments, and Modifications: 

Standards and measures are implemented by all local education agencies without 
modification. A monitoring and documentation system audits programs for compliance 

with the standards. 
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UTAH 



Type of System: 

Utah has developed separate systems of core performance standards and measures to 
evaluate secondary and postsecondary vocational education programs funded with 
Perkins money. While the systems contain the same measures and, in most instances, 
the same indicators, the measurement instrument and/or the standards may be unique 
to one level or the other. In addition, the systems contain both mandatory and optional. 

measures. 

Measures and Standards and Special Populations: 

The secondary and postsecondary measures are: 

access and accessibility, 
academic achievement, 
job/work skill attainment (optional), 
retention (optional), 

occupational competency gain/attainment (optional), and 
placement. 

Multiple indicators are used to measure access and accessibility. As an option, districts 
and institutions may choose to assess program completion in evaluating vocational 
programs. Mandatory indicators of access and accessibility include: 

awareness of vocational opportunities information, 

development of Student Education Occupation Plans (secondary only), 

enrollment (special populations), 

non-traditional program enrollment, and 

program completion. 



Awareness is measured on the comprehensiveness of the dissemination of information 
regarding vocational opportunities to parents and students in the 7th/8th and 12th 
grades The attainment standard is that 75% of all students and parents will receive 
this information and that 95% of special population students and parents wi>: receive it. 
By 1 994, the percentages are to be 85% and 95% respectively for 7th and 8th grade 
students and their parents and 85% and 99%, respectively, for 12th grade students and 

their parents. 

Development of Student Education Occupation Plans (SEOP) is an indicator at the 
secondary level only. Satisfactory development of SEOPs is demonstrated when 65% 
of all students and 100% of special population students have effective, functional 
SEOP's or Individualized Education Programs (lEPs) in place by the end of ninth grade. 



The growth standard is that 95% and 100% of students will have effective, operational 
SEOPs/IEPs by June 1996. 

Enrollment as an indicator of access and accessibility is measured on the percentage of 
special population students enrolled in approved program areas and on the extent of 
change in non-traditional enrollments. The percentage of special population 
enrollments is to be equal to or greater than the percentage of special populations 
students in the high school or college population. The change in non-traditional 
enrollments should increase annually over the funding period. Additionally, at the 
secondary level, these increases are to occur until effective, operational SEOPs are th6 
driving force for determining student enrollment. 

An optional indicator, program completion, is determined by comparing the completion 
rate of students in Perkins-funded programs with the completion rates of students in 
other comparable programs. The "Perkins-completers" rate of increase is 5% annually 
and should equal or exceed the "non-Perkins-completers" rate. 

Currently, two indicators of academic achievement are outlined in the secondary 
system. (1) state-wide Stanford Achievement testing of 8th and 11th grade students in 
mathematics, reading, and language artsand/or English, and (2) course completion. 

In using standardized testing as the measure of academic achievement, local education 
agencies are to establish both "acceptable" and "excellent" attainment standards in 
each academic area based upon national percentile ranks as well as three-year 
improvement targets. 

To demonstrate course completion by applied technology program completers: 

1 Schools may document completion of remedial instruction with a grade of 
"C" or better by 80% of those applied technology students requiring 
remedial instruction, 

2. Schools may document completion of Algebra I and Core English 10 (or 

the writing assignments required in Grades 9/10) with a grade of C or 

better by 70% of course completers, and 

3 Schools may document completion of advanced math and English 
courses with a grade of "C" or better by 70% of course completers. 

At the postsecondary level, course completion is used to measure achievement of 
academic skills. Basic skill attainment is measured on the percentage of course 
completers attaining a "C" in remedial or general basic education courses. Advanced 
skill attainment is measured on the percentage of course completers attaining a C in 
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an applied academic course above the 100's level. The minimum standard in both 
instances is 70%. 

Job or work skill attainment is an optional measure which secondary and 
postsecondary institutions may select. It is measured by determining the number of 
program completers who demonstrate mastery of program competencies either through 
portfolios or from a standardized test of workplace skills, (i.e., Work Keys). The 
standard for program completers with portfolios is 95% while the standard for mastery 
of all competencies via a standardized test is 90%. 

Retention is measured at both the secondary and postsecondary level by the number of 
students achieving positive outcomes (diploma, certificate, job placement) from their 
vocational program. Districts and institutions may determine standards and targets 
based upon baseline data for student retention and/or completion. In addition, at the 
secondary level, the state has set a 90% annual student advancement rate for 
measuring students' annual progress toward completion of their vocational program. 

At the secondary level, rates of occupational competency gain and attainment in data 
processing, word processing, accounting, workplace skills, apprenticeship readiness, 
and Automotive Service Excellence automotive programs are based upon state or 
industry-identified competencies. Schools report the rate at which students are 
meeting 80% of these state or industry competencies. For all other programs at the 
secondary level and all programs at the postsecondary level, the attainment standard is 
that 70% of vocational students master the expected competencies with a grade of "C" 
or better. The improvement standard (secondary only) establishes as a future goal that 
85% of all secondary vocational students will pass the expected competencies with a 
grade of "C+" or better. 

Placement, a required measure for both secondary and postsecondary programs, 
assesses the rate of training-related placement of program completers into jobs, further 
education, and military service. All program completers are to be identified and 
counted in the calculation. The placement rate, calculated on a minimum one month 
employment within nine months of program completion, was set initially at 45% with 
improvement to 65% by 1997. As an option, a district or institution may also assess 
employer and student satisfaction rates to achieve an 80% rate for both groups. 

Evaluation Process: 

Many of the current reporting sources for collecting data are generated manually. This 
information is entered into the State Vocational Education Management Information 
System (VEMIS), which resembles the former Vocational Education Data System 
(VEDS) report. 
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Future Activities: 



Future activities include the development of a computerized system to provide all 
required data and the continued development of financial incentives (from state funds) 
to reward districts that train students in marketable competencies resulting in 
immediate post-high school employment. 



VERMONT 



Type of System: 

Vermont has developed a singular system of performance standards and measures to 
evaluate all secondary and postsecondary vocational education programs 

Measures and Standards and Special Populations: 

The system contains multiple measures in each of four areas: 

student progress, 
program quality, 

access (special populations), and 
progress (special populations). 

Student progress pertains to student mastery of core competencies in the program 
area, mastery of workplace skills, academic gain, and placement of program 
completers. 

Entry-level mastery of 90% of the core competencies by 80% of the full time students 
exiting the program is the measurement of occupational competence. The standard for 
workplace skills is entry-level mastery by 80% of the full time students exiting the 
vocational program. 

Academic gain is measured by comparing students' average program-entry score with 
their average program completion score. The standardized test used is the Test of 
Adult Basic Education (TABE). The standard is an increase in average score. 

Placement is measured using one and three year follow-up studies of program 
completers to determine that at least 60% of program completers are pursuing 
"appropriate" employment or postsecondary education. Appropriate employment is 
employment related to the student's training program. In addition, data are collected on 
those employed in unrelated employment, unemployed and looking for employment, 
and program completers not in the labor market. 

Program quality is assessed by evaluating how well a program functions in eight areas, 
academic skills, occupational skills, workplace skills, capacity, linkages, relevance, 
special populations, and adults. In addition to using the results of the student 
performance measures described above, locals rate their programs as to how well they. 




integrate academic/vocational education, 

provide for sequential courses, 

provide experience in all aspects of the industry, 
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provide the necessary workplace competencies, 

provide current curriculum, instructional materials, equipment needs, 

provide programs relevant to the current/future needs of the workforce, 

establish linkages between secondary and postsecondary programs, 

meet the needs of special population students, and 

meet the training and retraining needs of adults. 

Measures of access and progress are used to identify the participation and success of 
special populations in vocational education programs. Access is determined by 
calculating the enrollment percentages of various target groups (students with 
individualized education programs (lEPs), educationally disadvantaged, English as a 
second language (ESL), and male/female ratio) in the grade, technical center, and 
program. Progress is measured by identifying. 

the number of students completing the requirements of their IEP, 
the number of economically and/or educationally disadvantaged students 
achieving the occupational competency and workplace skill standards 
established for all students, 

the number of disadvantaged and/or non-traditional students demonstrating 
academic gain on a pre-post program assessment using the Test of Adult 
Basic Education (TABE), and 

the percentage of disabled, disadvantaged, and non-traditional students placed 
in appropriate employment or education. 

Evaluation Process: 

Each year, programs are required to perform end-of-year self-evaluations using the 
standards and measures adopted by the State Board. The results of each program s 
evaluation are forwarded to the State. By 1995, all standards will be applied to all 
exiting students and recipients will have fully implemented all measures and standards 

Incentives, Adjustments, and Modifications: 

Adjustments are provided for the student placement measure through weighting. An 
additional one percent is added to the placement percentages for each disadvantaged 
or non-traditional student and an additional 1.5% is added for each handicapped 
student in related employment or postsecondary education. 
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VIRGINIA 



Type of System: 

Virginia has established a singular system of program performance standards and 
measures for evaluating secondary and postsecondary vocational education programs 
supported by federal, state, and local funds. 

Measures and Standards and Special Populations: 

The system includes four measures: 

academic achievement, 
occupational competence, 

access to and success in vocational education (special populations), and 
successful transition. 

Academic achievement will eventually be measured by identifying the gain in basic and 
advanced skills in mathematics, reading, and communication demonstrated by 
vocational completers on criterion-referenced tests. Until a reliable criterion-referenced 
test is available, however, LEA’s are to utilize course grades, grade point averages, 
completion or failure rates, scores from the norm-referenced Test of Achievement and 
Proficiency (TAP), any other standardized norm-referenced test, or locally developed 
criterion-referenced tests of academic performance. At the postsecondary level, 
academic achievement is determined by identifying the gain on a standardized or 
locally developed academic measure or by measuring the successful completion of 
academic courses required in any occupational-technical education curriculum. No 
state standards are established. 

Occupational competency is defined as occupational-technical and employability 
knowledge and skills. Occupational competence is measured by determining the 
percentage of vocational completers mastering the occupational competencies equal to 
or exceeding those required for entrance into career-sustaining employment, approved 
apprenticeship programs, or appropriate postsecondary education as determined by a 
locally validated competency list reviewed by the local vocational advisory committee. 
The standard is for 80% of vocational completers to master 80% of the competencies. 



Access to vocational education programs by members of targeted groups is determined 
at the secondary level by local education agencies demonstrating full compliance with 
the incentives and adjustments required under the Perkins Act and other related 
legislation. At the postsecondary level, institutions are to demonstrate that the 
representation of identified target groups and special populations in occupational- 
technical programs of study is proportional to their membership in relevant community 

college populations. 



For both secondary and postsecondary programs, success in vocational education is 
measured by applying the academic achievement and occupational competence 
measures to targeted special population groups. 

At the secondary level, successful transition is determined by the placement of a 
vocational completer in a full-time activity (paid employment, related or unrelated, 
homemaking, postsecondary or continuing education , military or voluntary 
govemment/community service) or part-time combinations of these activities totaling 
full-time equivalency. The standard is that 90% of secondary vocational program 
completers will make the transition successfully. 

At the postsecondary level, successful transition is defined as the placement of a 
occupational completer/graduate in a full-time activity (educationally-related 
employment, fulfilling upgrading or retraining goals, transition to an associate degree, 
or baccalaureate degree program, transfer to a four-year postsecondary program, 
participation in an apprenticeship program, or enlistment into military or government, or 
community service) or the full-time equivalency of combinations of the above activities. 
The rate of successful transition is 80%. 

Evaluation Process: 

Each local education agency or postsecondary institution completes at the close of 
each school year, a performance report summarizing their efforts towards 
accomplishing each of the four performance measures. Those programs not meeting 
one or more of the standards prepares a program improvement plan. The report and 
plans are forwarded to the State Department of Education by July 1 of each year. The 
reports are reviewed and evaluated. A university-based contractor conducts data 
collection, analyzes local report data, prepares reports, and provides technical 
assistance related to evaluation procedures. 

Future Activities: 

A fifth measure is proposed: Employer and employee satisfaction with vocational 
preparation. Plans are to develop and implement this measure during school year 
igg 4 _g 5 (after full review by the Committee of Practitioners and approval of the State 
Board of Education). 
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WASHINGTON 



Type of System: 

Separate systems of performance standards and measures are developed to evaluate 
all secondary and postsecondary vocational education programs in the state. 

Measures and Standards: 

The four secondary level measures are: 

basic skill competencies, 

work maturity skill competencies, 

entry-level occupational competencies, and 

skill competencies to progress in education, training, employment. 

Basic skill competencies are measures of students' ability to demonstrate mastery of 
the following: reading, writing, computing basic math computations, understanding and 
communicating in writing and orally, working well with others including human relation 
skills and multicultural sensitivities, and performing critical thinking including problem- 
solving, decision-making, organizing and prioritizing. 

Work maturity skills are measures of students' ability to demonstrate acquisition or 
mastery of the following: possession of appropriate work ethic, appearance, and 
attitude toward work performance, ability to lead and follow appropriately, ability to 
adapt skills and attitudes to change and to lifelong learning, ability to manage personal 
and family life, ability to identify and set education and employment goals, and the 
possession of job search skills and knowledge of how to access job information. 

Entry-level occupational skills are measures of students', knowledge and 
comprehension of all aspects of the industry being prepared to enter, transition to work 
within the field, safety precautions, expectations of the world of work, technology, and 
skill mastery of specific occupational training. 

The skill competencies that demonstrate students' ability to progress in further 
education training, or employment include the achievement of education objectives, 
satisfaction of the student, instructor, and employer on all measures, and employment 
or enrollment in the next level of education. 

At the secondary level, assessment methodology of the measures and standards are 
determined locally through collaborative efforts with local advisory committees. 



The five postsecondary measures are: 



class completion, 
program proficiency, 
placement, 

equal access — community college (special populations), and 
equal access - technical college (special populations). 

Class completion measures completion rates for classes comprising state approved 
programs which designate student competency with certificates, degrees, or 
competency lists. The standard is an 80% class completion rate. The cohort is 
students in their second academic period (quarter, semester) of enrollment. In classes 
utilizing grades to measure competency, the standard is 80% class completion with a 
grade of "C" or better. 

Program proficiency is measured by the number of state approved postsecondary 
vocational technical programs having developed descriptions of program proficiencies. 
Proficiencies are to identify the skills and skill levels to be attained by program 
completers in communication, computation, human relations, and occupationally 
specific skills. The standard is that 100% of all programs have these descriptions. 

Placement is a measure of the percentage of program completers who are employed, 
employed in jobs related to their training, in further education or training, in military 
service or other service such as the Peace Corps. 

Equal access in the community college is measured by the percentage of special 
population students enrolled in each vocational program compared to the percentage of 
special population students in the institution’s non-vocational population. The ratio is 
1.0 or greater. Equal access in the technical college is measured by the percentage of 
special population students enrolled in each vocational program as compared with the 
percentage of special population adults residing in the technical college's county. 
Enrollment of special population students is to be equal to or greater than the 
percentages represented in the county adult population. 

Assessment methodology is a local decision. 

Special Populations: 

Special populations are addressed in the postsecondary system by the development of 
a measure of equal access (see above). 
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Evaluation Process: 



At the secondary level, school districts use a "gap analysis" process to measure 
progress towards meeting the standards. A survey instrument, distributed to 
administrators, teachers, advisory committee members, and students, requests 
respondents to indicate how often students should be able to demonstrate certain 
competencies and how often students are able to demonstrate competencies. The gap 
between the perception of where students are and where they should be in mastery of 
competencies is the measure of how well the school is preparing students. State 
guidelines are that performance should be at least 80 % and performance levels below 
65 % need improvement, however, satisfactory accomplishment of a measure is a local 
determination. If a district determines that a standard is not met, a local improvement 
plan is developed and forwarded as part of the local application. 

At the postsecondary level, evaluation is a local initiative. Local institutions conduct 
formative and summative evaluations on individual courses where the emphasis is on 
mastery of competencies. Program evaluations, conducted annually, emphasize 
continuous improvement with local determination of the most effective and efficient 
method for evaluation and implementing changes. 

Future Activities: 

At the postsecondary level, the State will be designing an accountability system for all 
workforce training and education. Review of the current measures and standards, as 
well as current evaluation and monitoring activities, will be a part of this effort. 
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WEST VIRGINIA 



Type of System: 

West Virginia has established a singular system of program performance measures 
and standards to evaluate all vocational programs in the state. 

Measures and Standards and Special Populations: 

The six measures are: 

academic achievement, 
instructional competencies, 
enrollment (special populations), 
placement, 

employer satisfaction, and 
student satisfaction. 

Academic achievement measures the reading and math levels of occupational program 
completers Student test levels will be at or above the level required for the 
occupational levels as established by the ACT/Work Keys. Students with Individualized 
Education Programs (lEPs) indicating they cannot take standardized tests such as 
WorkKeys are to be exempted from the WorkKeys assessment and provided alternative 

assessments. 

The measure of instructional competencies assesses the rate at which program 
completers score a minimum of 80% on a state-adopted core content test or, if a test is 
not available, using goal profiles. Goal profiles are based on state-approved 
instructional objectives. The standard is that the average program completer will score 
a minimum of 80% on the content test. Licensed Practical Nursing, Cosmetology, and 
the Respiratory Therapy programs are to use licensure testing in lieu of the content 
test The standard for these programs is that 80% of program completers will pass the 
licensing examination. Students with lEPs may be exempt from standardized content 
exams and assessed against the instructional objectives outlined in their IEP. 

Enrollment measures the rate at which special population students are enrolled in 
vocational programs as compared with the percentage of all special populations 
students enrolled in the general school population. The standard is that the percentage 
of special population students being served in vocational programs be greater than the 
percentage of all special populations students in the general population of the local 
educational agency. 

Placement measures the post program activities of program completers within one year 
after leaving the program. At least 85% of the program completers will be employed, in 
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the military full-time, or continuing their education. Of those employed, at least 65% will 
be employed in the field or related area for which they are trained 

Employer satisfaction assesses the extent to which employers of program completers 
are satisfied with their performance within one year of their employment as rated by 
80% of employers on a state-adopted survey instrument. Characteristics surveyed 
include technical knowledge, safety habits, responsibility, communications skills, 
decision-making, attitude, teamwork, attendance and grooming, customer relations, 
and potential for advancement. Average employee ratings will be 3.0 or more on a 4.0 
scale for each occupational program. 

Student satisfaction assesses students' evaluation of the quality and effectiveness of 
the instructional offering 30 days prior to program completion. Program elements 
surveyed include teacher's subject knowledge and instructional ability, emphasis on 
math and science, amount, condition and quality of classrooms, labs, books, 
equipment, tools, and supplies, opportunity for classroom participation, and perception 
of job opportunities. The standard is that the average student rating will be 2.5 or more 
on a 4.0 scale for each program. 

Evaluation Process: 

Local education agencies conduct annual evaluations using state-developed report 
forms. Results (by program) are forwarded to the state each year by June 30th. 

Incentives, Adjustments, and Modifications: 

Local recipients may make modifications to the system. Requests are reviewed by the 
state to assure they conform to the assessment criteria established for the state-wide 
system prior to approval or rejection. 



WISCONSIN 



Type of System: 

Wisconsin has established separate systems of performance standards and measures 
for all secondary and postsecondary programs receiving Perkins funding. 

Measures and Standards and Special Populations: 

The five secondary measures are: 

academic skill, 

graduation, 

placement, 

participation (special populations), and 
participation (gender equity). 

Academic skill is evidenced by the vocational education student s successful 
completion of the credits required for graduation from high school. To demonstrate 
compliance, Districts will require that vocational education participants fulfill the same 
state-wide graduation requirements established by State statute for all students. 

Graduation rate is calculated to determine that vocational students have a graduation 
rate equal to or higher than the graduation rate for a district's total student population. 
The gain standard established is that vocational education participation graduation 
rates will increase by a factor of 1 .05 each year until a 95% rate is achieved. 

Placement is measured by comparing the rate of placements in employment, further 
education, and military service related to the instructional program area with the rate of 
placements that are unrelated to the instructional program area. The gain standard is 
that the district will demonstrate an increase in its rate of related vocational education 
graduates placements by a factor of 1 .05 each year until a 90% rate is achieved. 

Rate of participation of special populations is compared against the rate of participation 
of a district's general population in vocational education. Districts are to increase the 
rate of participation of special populations in vocational education by a factor of 1 .05 
each year until the rate is equal to or greater than the participation rate of a district's 
general population, in vocational education. 

Participation is also measured for gender equity. A district is to increase its participant 
rate by either gender within an instructional program by a factor of 1 .05 over each 
preceding year’s rate until the rate is greater than the participant rate of 25% for each 
gender within the instructional program. 



The postsecondary level measures in use are. 

program graduates, 
job placement, 

grant activity — positive training outcomes, and 
remedial basic skills. 

Program graduates and employment results are reported for all one and two year 
occupational programs. Employment results are based on graduates securing 
employment within 6 months of the end of the school year. The state standards are 
35% and 75%, respectively. 

The measure, grant activity — positive training outcomes, calculates the percentage of 
students receiving Perkins supported activities who achieve one or more training 
outcomes. Examples of possible training outcomes include "enrolled in an 
occupational program/course” and "entered a training program in another institution". 
The standard is that 40% of students receiving support services will achieve positive 
training outcomes. 

Recently implemented, remedial basic skills measures the percentage of students who 
were in a remedial basic skills class and subsequently completed at least one 
occupational course. 

Evaluation Process: 

At the secondary level, all the measures dovetail into the existing secondary vocational 
data system The district submits data to the state via the Vocational Education 
Enrollment Reporting System (VEERS). A district profile is developed which compares 
a district’s current year's performance with the previous year's performance (baseline 
data is FY 92). Any deficiencies are identified. Local agencies are to respond to the 
deficiencies by developing and implementing an improvement plan This plan is 
submitted to the state for review and approval within 45 days. 

At the postsecondary level, data are reported to the state via the Client Reporting 
System. Local institutions are to use the results of the performance measures to 
develop local institution plans and the local improvement plans required as part of the 
local application. 

Incentives, Adjustments, and Modificaions: 

Local educational agencies not achieving the established standard may request a 
modification to the standard due to conditions or factors outlined in the Act. 
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WYOMING 



Type of System: 

Wyoming has developed separate systems of performance standards and measures to 
evaluate secondary and postsecondary programs receiving Perkins funds. 



Measures and Standards and Special Populations. 
The secondary system includes the following measures: 



academic gain, 
competency gain, 
competency attainment, 
high school completion, 
placement, 

enrollment (special populations), and 
supplemental services. 



Academic gain is determined by the percentage of students demonstrating gain in math, 
reading, and language arts and/or science as measured by a performance assessment or 
other assessment tool. The standard is that eighty-five percent of the students enrolled in 
Perkins-assisted programs will demonstrate significant gain. Significant gain is 
determined for each school based upon each district's performance standards. 



Competency gain is determined by the percentage of students that demonstrate mastery 
of skills identified in Wyoming's Common Core of Knowledge and Common Core of SkiNs. 
These are coordinated with the SCAN'S competencies and foundation skills. The standard 
is that 85% of students in Perkins assisted programs will demonstrate a gain in 
competencies that are relevant to the Common Core of Knowledge and Skills. 

Competency attainment is determined by the percentage of vocational course completers 
passing a minimum of 80% of the occupational competencies designated on course 
competency checklists for the course. Eighty-five percent of students enrolled in Perkins 
assisted courses will satisfactorily complete the course. 



Graduation rate is calculated by the percentage of vocational students in Perkins-assisted 
programs earning a high school diploma or GED equivalent. The rate will equal or 
exceed that of the total school population and will be 90%. 



Placement is determined by the number of placements in additional training or education, 
military service, or employment compared to the total number of students reporting on a 
follow-up survey. The standard is that 85% of those returning the survey will report 
placement in further education or training, military service, or employment. 



The established standard for the enrollment of special population students served in 
Perkins-assisted vocational education programs is equivalent to the percentage of special 
population students in the total school population of each district or as nearly as 
demographics allow. Supplementary services will be documented in each student's 
individualized Career Development Plan. 

The postsecondary measures include: 

learning and competency gain, 
competency attainment, 
placement, 

employer satisfaction, and 
supplementary services (special populations) 

Postsecondary and adult programs measure student learning and competency gain by 
calculating the percentage of students in Perkins assisted programs demonstrating gain 
in competencies that are relevant to the SCANS competencies and skills. The standard 
is that 85% of students demonstrate gain. Data are aggregated to institution level. 

Competency attainment is determined by the percentage of postsecondary program 
students passing 80% of the competencies in workplace skills as enumerated in SCANS 
and a minimum of 80% of competencies designated for a course which have been 
validated by each program's Advisory Committee. Institutions may select their choice of 
evaluation methods. 

Placement is defined as the employment, further education or military service enlistment 
of program completers. The standard is that 85% of all program completers are placed 
as identified through each institution's student followup procedures 

Employer satisfaction is determined by the percentage of satisfactory ratings given to 
program completers by their employers as reported on employer surveys. Eighty-five 
percent of completers employed in areas related to their training will receive satisfactory 
ratings of their occupational skills and competencies. 

Incentives, Adjustments and Modifications: 

Supplementary services are viewed as an incentive by encouraging program accessibility 
to special population students. The standard is that all students performing below the 
criteria for each vocational program will be afforded the opportunity to receive 
supplementary services. 

Evaluation Process: 

Lo ca l agencies are provided the standards and measures to use as local guidelines for 
evaluation purposes. In 1992-93 the state assisted locals by incorporating the standards 





118 



and measures into the State assessment. Individualized data were provided to 
school district and community college for them to evaluate their own progress. 
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